2024-03-24T11:19:42.862 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T11:19:42.862 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921 --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921/orig.config.yaml 2024-03-24T11:19:42.892 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:19:42.983 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617921' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwm6wqBFZn2UGMeaF0VzicBXxM770gis4JyqHWgkTzsetR6P4i6XzU2gZ24EJurcNCDSUNkyLuGL4k3bqBqBv4= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDlHgECTRqX4sRlkcUa7B1Dst/KYJZg6YqRHj/fr8QWzXPf0nBFRBaEVO9zCS0ft+pjxZw8Uf+xe1owZwW2Ikrw= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T11:19:42.984 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T11:19:42.984 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T11:19:42.984 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T11:19:42.986 INFO:teuthology.task.internal:Checking packages... 2024-03-24T11:19:43.012 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T11:19:43.012 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:19:43.012 INFO:teuthology.packaging:ref: None 2024-03-24T11:19:43.012 INFO:teuthology.packaging:tag: None 2024-03-24T11:19:43.012 INFO:teuthology.packaging:branch: squid 2024-03-24T11:19:43.012 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:19:43.012 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T11:19:43.229 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T11:19:43.231 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T11:19:43.238 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T11:19:43.238 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T11:19:43.328 INFO:teuthology.task.internal:Saving configuration 2024-03-24T11:19:43.339 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T11:19:43.373 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:19:43.400 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:14:12.602723', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwm6wqBFZn2UGMeaF0VzicBXxM770gis4JyqHWgkTzsetR6P4i6XzU2gZ24EJurcNCDSUNkyLuGL4k3bqBqBv4='} 2024-03-24T11:19:43.421 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:14:12.600968', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDlHgECTRqX4sRlkcUa7B1Dst/KYJZg6YqRHj/fr8QWzXPf0nBFRBaEVO9zCS0ft+pjxZw8Uf+xe1owZwW2Ikrw='} 2024-03-24T11:19:43.421 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T11:19:43.428 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T11:19:43.428 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-24T11:19:43.428 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T11:19:43.521 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f836991f0d0>, signals=[15]) 2024-03-24T11:19:43.521 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T11:19:43.527 INFO:teuthology.task.internal:Opening connections... 2024-03-24T11:19:43.528 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-03-24T11:19:43.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:43.622 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-03-24T11:19:43.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:19:43.713 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T11:19:43.719 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-03-24T11:19:43.736 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-03-24T11:19:43.737 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="9" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="9" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:19:43.793 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:19:43.794 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:19:43.794 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:19:43.794 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:19:43.794 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-03-24T11:19:43.821 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-03-24T11:19:43.839 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-03-24T11:19:43.839 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-03-24T11:19:43.895 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="9" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="9" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:19:43.896 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:19:43.897 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-03-24T11:19:43.919 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T11:19:43.929 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T11:19:44.028 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T11:19:44.028 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:19:44.030 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:19:44.046 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T11:19:44.070 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T11:19:44.071 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:19:44.087 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:19:44.180 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T11:19:44.234 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T11:19:44.251 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T11:19:44.251 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T11:19:44.252 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:19:44.252 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:19:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:19:44.268 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-24T11:19:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:19:44.269 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-03-24T11:19:44.325 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:19:44.325 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.14.0-361.el9.x86_64 2024-03-24T11:19:44.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2024-03-24T11:19:44.327 INFO:teuthology.orchestra.run.smithi149.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:19:44.328 INFO:teuthology.task.kernel:Running kernel on smithi149: 5.14.0-361.el9.x86_64 2024-03-24T11:19:44.328 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2024-03-24T11:19:47.006 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 6.4 MB/s | 6.7 MB 00:01 2024-03-24T11:19:47.034 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.7 MB 00:00 2024-03-24T11:19:48.768 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-24T11:19:48.939 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 2.4 MB/s | 2.0 MB 00:00 2024-03-24T11:19:51.505 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-03-24T11:19:51.517 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-03-24T11:19:53.533 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-24T11:19:57.382 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.1 MB 00:00 2024-03-24T11:19:58.966 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-24T11:20:00.600 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T11:20:05.550 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-24T11:20:07.665 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:20:07.665 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:20:07.800 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-24T11:20:07.808 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-24T11:20:07.808 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-03-24T11:20:07.808 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-24T11:20:07.808 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-24T11:20:07.808 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:20:07.808 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-24T11:20:07.808 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 76 M 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 123 M 2024-03-24T11:20:07.809 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-24T11:20:08.142 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 203 kB/s | 44 kB 00:00 2024-03-24T11:20:09.210 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-24T11:20:09.718 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 17 MB/s | 27 MB 00:01 2024-03-24T11:20:09.901 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.5 MB/s | 15 MB 00:01 2024-03-24T11:20:09.904 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:20:09.904 INFO:teuthology.orchestra.run.smithi145.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-24T11:20:10.567 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-24T11:20:10.936 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-24T11:20:10.937 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-24T11:20:11.955 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-24T11:20:11.955 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-24T11:20:13.940 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - AppStream 903 kB/s | 19 MB 00:21 2024-03-24T11:20:14.413 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-24T11:20:15.322 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:20:15.410 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:20:18.147 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:20:18.280 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:20:18.685 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.1 MB 00:01 2024-03-24T11:20:20.291 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-24T11:20:21.745 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-24T11:20:22.980 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:20:23.088 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:20:26.826 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-24T11:20:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:20:28.958 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:20:28.958 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:20:29.052 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-24T11:20:29.059 INFO:teuthology.orchestra.run.smithi149.stdout:Install 4 Packages 2024-03-24T11:20:29.060 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:20:29.060 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 76 M 2024-03-24T11:20:29.060 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 123 M 2024-03-24T11:20:29.060 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-03-24T11:20:29.469 INFO:teuthology.orchestra.run.smithi149.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 128 kB/s | 44 kB 00:00 2024-03-24T11:20:41.485 INFO:teuthology.orchestra.run.smithi149.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.2 MB/s | 15 MB 00:12 2024-03-24T11:20:51.050 INFO:teuthology.orchestra.run.smithi149.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.5 MB/s | 33 MB 00:21 2024-03-24T11:20:51.352 INFO:teuthology.orchestra.run.smithi149.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.3 MB/s | 27 MB 00:21 2024-03-24T11:20:51.352 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:20:51.352 INFO:teuthology.orchestra.run.smithi149.stdout:Total 3.4 MB/s | 76 MB 00:22 2024-03-24T11:20:52.016 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-03-24T11:20:52.381 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-03-24T11:20:52.381 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-03-24T11:20:53.406 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-03-24T11:20:53.406 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-03-24T11:20:55.887 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-03-24T11:20:56.505 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:20:56.684 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:20:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:20:56.921 INFO:teuthology.orchestra.run.smithi145.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:20:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:20:56.965 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:20:57.342 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:20:57.343 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:20:57.343 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:20:57.343 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:20:57.746 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:20:57.746 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:20:57.746 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-24T11:20:57.746 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:20:57.746 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:20:57.747 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:20:57.747 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:20:57.747 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:20:57.747 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-24T11:20:58.087 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:20:58.188 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:20:58.188 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:20:58.188 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:20:58.188 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:20:58.188 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:20:58.188 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T11:20:58.189 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:20:58.189 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:20:58.267 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:20:58.268 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:20:58.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2024-03-24T11:20:58.301 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2024-03-24T11:20:58.301 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2024-03-24T11:20:58.301 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.06 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 68.el9 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Group : Unspecified 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 8264254 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:20:58.302 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : builder@centos.org 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:20:58.303 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:20:58.304 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:20:58.304 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:20:58.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:20:58.931 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-03-24T11:20:59.472 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:20:59.603 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:21:00.346 INFO:teuthology.orchestra.run.smithi145.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:21:00.374 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-03-24T11:21:00.377 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-24T11:21:00.392 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.4unbt3fZOc 2024-03-24T11:21:00.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4unbt3fZOc 2024-03-24T11:21:00.461 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.4unbt3fZOc 2024-03-24T11:21:00.599 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.4unbt3fZOc is 6KB 2024-03-24T11:21:00.647 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.4unbt3fZOc 2024-03-24T11:21:00.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:21:00.725 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:21:00.725 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:21:00.725 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:21:00.725 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:21:00.727 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:21:00.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-03-24T11:21:04.294 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:21:04.395 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:21:09.264 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:21:30.880 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T11:21:30.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-24T11:21:30.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:21:38.755 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:21:38.755 INFO:teuthology.orchestra.run.smithi149.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:21:38.755 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:21:38.799 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:21:39.150 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:21:39.150 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:21:39.150 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:21:39.150 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:21:39.577 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:21:39.577 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:21:39.578 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-03-24T11:21:39.578 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:21:39.578 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:21:39.578 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:21:39.578 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:21:39.578 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:21:39.578 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-03-24T11:21:39.895 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:21:40.004 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:21:40.005 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:21:40.005 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:21:40.005 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:21:40.005 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:21:40.005 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T11:21:40.005 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi149.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:21:40.005 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:21:40.086 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:21:40.087 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:21:40.087 DEBUG:teuthology.orchestra.run.smithi149:> sudo rpm -qi grub2-tools 2024-03-24T11:21:40.130 INFO:teuthology.orchestra.run.smithi149.stdout:Name : grub2-tools 2024-03-24T11:21:40.130 INFO:teuthology.orchestra.run.smithi149.stdout:Epoch : 1 2024-03-24T11:21:40.130 INFO:teuthology.orchestra.run.smithi149.stdout:Version : 2.06 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Release : 68.el9 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Architecture: x86_64 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Group : Unspecified 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Size : 8264254 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:License : GPLv3+ 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Packager : builder@centos.org 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:Vendor : CentOS 2024-03-24T11:21:40.131 INFO:teuthology.orchestra.run.smithi149.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout:Summary : Support tools for GRUB. 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout:Description : 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout:hardware devices. 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:21:40.132 INFO:teuthology.orchestra.run.smithi149.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:21:40.133 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:21:40.133 INFO:teuthology.task.kernel:Updating grub on smithi149 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:21:40.133 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:21:40.724 INFO:teuthology.orchestra.run.smithi149.stderr:Generating grub configuration file ... 2024-03-24T11:21:42.050 INFO:teuthology.orchestra.run.smithi149.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:21:42.089 INFO:teuthology.orchestra.run.smithi149.stderr:done 2024-03-24T11:21:42.090 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-03-24T11:21:42.105 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.tgz8xj6tPF 2024-03-24T11:21:42.105 DEBUG:teuthology.orchestra.run.smithi149:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tgz8xj6tPF 2024-03-24T11:21:42.188 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.tgz8xj6tPF 2024-03-24T11:21:42.329 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.tgz8xj6tPF is 6KB 2024-03-24T11:21:42.377 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.tgz8xj6tPF 2024-03-24T11:21:42.391 DEBUG:teuthology.orchestra.run.smithi149:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:21:42.456 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:21:42.456 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:21:42.456 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:21:42.456 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:21:42.457 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:21:42.617 DEBUG:teuthology.orchestra.run.smithi149:> sudo shutdown -r now 2024-03-24T11:21:49.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-24T11:21:58.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-24T11:21:58.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:01.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-24T11:22:12.630 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T11:22:12.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-24T11:22:12.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:13.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-24T11:22:13.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:31.074 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-03-24T11:22:31.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-24T11:22:40.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-24T11:22:40.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:43.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-03-24T11:22:46.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-24T11:22:46.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:22:47.306 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-24T11:22:47.691 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-24T11:22:47.691 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:22:47.692 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-24T11:22:47.705 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:22:47.705 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:22:47.705 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:22:47.705 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:22:48.707 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T11:22:48.707 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:22:48.761 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2024-03-24T11:22:48.802 DEBUG:teuthology.parallel:result is None 2024-03-24T11:22:55.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-24T11:22:55.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:23:55.145 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:24:10.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-24T11:24:10.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:24:10.436 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-03-24T11:24:10.807 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-24T11:24:10.807 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:24:10.807 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-03-24T11:24:10.822 INFO:teuthology.orchestra.run.smithi149.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:24:10.823 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:24:10.823 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:24:10.823 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:24:11.824 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T11:24:11.824 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:24:11.891 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-03-24T11:24:11.925 DEBUG:teuthology.parallel:result is None 2024-03-24T11:24:11.926 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:24:11.933 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:24:11.933 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:24:11.936 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:24:11.954 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:24:12.041 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:24:12.071 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:24:12.071 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:24:12.074 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:24:12.108 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:24:12.136 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:24:12.136 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:24:12.142 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:24:12.180 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:24:12.192 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:24:12.194 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:24:12.207 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:24:12.208 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:24:12.215 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:24:12.215 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:24:12.238 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:24:12.268 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:24:12.278 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:24:12.279 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:24:12.308 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:24:12.326 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:24:12.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:24:12.456 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:24:12.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:24:12.586 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:24:12.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:24:12.653 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:24:12.697 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:24:12.761 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:24:12.804 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:24:12.848 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-24T11:24:12.849 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:24:12.918 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-03-24T11:24:12.920 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-03-24T11:24:12.966 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:24:13.004 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:24:13.401 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:24:13.410 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:24:13.410 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:24:13.472 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:24:13.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2024-03-24T11:24:13.591 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2024-03-24T11:24:13.591 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2024-03-24T11:24:13.662 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2024-03-24T11:24:13.663 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:24:13.664 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2024-03-24T11:24:13.682 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2024-03-24T11:24:13.682 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2024-03-24T11:24:13.704 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2024-03-24T11:24:13.705 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi145.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:24:13.705 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:24:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.750 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:24:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.778 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:24:13.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:24:13.834 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:24:13.874 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:24:13.886 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T11:24:13.891 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T11:24:14.028 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:24:14.047 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:24:14.048 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi145.front.sepia.ceph.com,smithi149.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:28:27.608 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi145.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com')] 2024-03-24T11:28:27.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-24T11:28:27.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:28:27.706 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-24T11:28:27.781 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-24T11:28:27.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-24T11:28:27.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:28:27.874 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-03-24T11:28:27.950 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-24T11:28:27.951 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:28:27.961 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:28:27.961 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:28:27.962 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:28:27.964 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:28:27.964 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:28:27.993 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:28:28.010 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:28:28.022 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:28:28.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2024-03-24T11:28:28.040 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:28:28.052 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2024-03-24T11:28:28.069 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2024-03-24T11:28:28.069 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:28:28.083 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2024-03-24T11:28:28.085 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:28:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:28:28.116 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:28:28.136 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:28:28.140 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:28:28.140 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-24T11:28:28.140 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.140 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.140 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.140 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.167 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:28:28.170 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:28:28.170 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-03-24T11:28:28.171 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.171 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.171 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.171 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:28:28.172 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:28:28.268 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:28:28.268 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-24T11:28:28.269 INFO:tasks.cephadm:Cluster fsid is a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:28:28.269 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:28:28.269 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.145', 'mon.c': '[v2:172.21.15.145:3301,v1:172.21.15.145:6790]', 'mon.b': '172.21.15.149'} 2024-03-24T11:28:28.269 INFO:tasks.cephadm:First mon is mon.a on smithi145 2024-03-24T11:28:28.269 INFO:tasks.cephadm:First mgr is y 2024-03-24T11:28:28.269 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:28:28.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo hostname $(hostname -s) 2024-03-24T11:28:28.298 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname $(hostname -s) 2024-03-24T11:28:28.327 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-24T11:28:28.327 DEBUG:teuthology.orchestra.run.smithi145:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:28:28.545 INFO:teuthology.orchestra.run.smithi145.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 24 11:28 /home/ubuntu/cephtest/cephadm 2024-03-24T11:28:28.546 DEBUG:teuthology.orchestra.run.smithi149:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:28:28.719 INFO:teuthology.orchestra.run.smithi149.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 24 11:28 /home/ubuntu/cephtest/cephadm 2024-03-24T11:28:28.720 DEBUG:teuthology.orchestra.run.smithi145:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:28:28.735 DEBUG:teuthology.orchestra.run.smithi149:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:28:28.761 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-24T11:28:28.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-24T11:28:28.778 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-24T11:28:29.582 INFO:teuthology.orchestra.run.smithi149.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:28:29.626 INFO:teuthology.orchestra.run.smithi145.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:28:46.542 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:28:46.542 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-24T11:28:46.542 INFO:teuthology.orchestra.run.smithi145.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-24T11:28:46.542 INFO:teuthology.orchestra.run.smithi145.stdout: "repo_digests": [ 2024-03-24T11:28:46.542 INFO:teuthology.orchestra.run.smithi145.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-24T11:28:46.543 INFO:teuthology.orchestra.run.smithi145.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-24T11:28:46.543 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-24T11:28:46.543 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:28:49.306 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-03-24T11:28:49.306 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-24T11:28:49.306 INFO:teuthology.orchestra.run.smithi149.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-24T11:28:49.306 INFO:teuthology.orchestra.run.smithi149.stdout: "repo_digests": [ 2024-03-24T11:28:49.306 INFO:teuthology.orchestra.run.smithi149.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-24T11:28:49.306 INFO:teuthology.orchestra.run.smithi149.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-24T11:28:49.306 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-03-24T11:28:49.307 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-03-24T11:28:49.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph 2024-03-24T11:28:49.360 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph 2024-03-24T11:28:49.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /etc/ceph 2024-03-24T11:28:49.430 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /etc/ceph 2024-03-24T11:28:49.457 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:28:49.458 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-24T11:28:49.458 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:28:49.458 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:28:49.458 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-24T11:28:49.458 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:28:49.458 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:28:49.458 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:28:49.459 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:28:49.459 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:28:49.459 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:28:49.459 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:28:49.486 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a2bb64be-e9d1-11ee-95cd-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-24T11:28:49.487 DEBUG:teuthology.orchestra.run.smithi145:mon.a> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T11:28:49.529 DEBUG:teuthology.orchestra.run.smithi145:mgr.y> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T11:28:49.571 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:28:49.571 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.145 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:28:49.765 INFO:teuthology.orchestra.run.smithi145.stderr:-------------------------------------------------------------------------------- 2024-03-24T11:28:49.765 INFO:teuthology.orchestra.run.smithi145.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'a2bb64be-e9d1-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.145', '--skip-admin-label'] 2024-03-24T11:28:49.788 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: 4.9.3-dev 2024-03-24T11:28:49.790 INFO:teuthology.orchestra.run.smithi145.stderr:Verifying podman|docker is present... 2024-03-24T11:28:49.812 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: 4.9.3-dev 2024-03-24T11:28:49.814 INFO:teuthology.orchestra.run.smithi145.stderr:Verifying lvm2 is present... 2024-03-24T11:28:49.814 INFO:teuthology.orchestra.run.smithi145.stderr:Verifying time synchronization is in place... 2024-03-24T11:28:49.821 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:28:49.827 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: inactive 2024-03-24T11:28:49.833 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: enabled 2024-03-24T11:28:49.839 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: active 2024-03-24T11:28:49.840 INFO:teuthology.orchestra.run.smithi145.stderr:Unit chronyd.service is enabled and running 2024-03-24T11:28:49.840 INFO:teuthology.orchestra.run.smithi145.stderr:Repeating the final host check... 2024-03-24T11:28:49.861 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: 4.9.3-dev 2024-03-24T11:28:49.864 INFO:teuthology.orchestra.run.smithi145.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:28:49.864 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl is present 2024-03-24T11:28:49.864 INFO:teuthology.orchestra.run.smithi145.stderr:lvcreate is present 2024-03-24T11:28:49.870 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:28:49.876 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: inactive 2024-03-24T11:28:49.882 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: enabled 2024-03-24T11:28:49.888 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: active 2024-03-24T11:28:49.889 INFO:teuthology.orchestra.run.smithi145.stderr:Unit chronyd.service is enabled and running 2024-03-24T11:28:49.889 INFO:teuthology.orchestra.run.smithi145.stderr:Host looks OK 2024-03-24T11:28:49.889 INFO:teuthology.orchestra.run.smithi145.stderr:Cluster fsid: a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:28:49.889 INFO:teuthology.orchestra.run.smithi145.stderr:Acquiring lock 139785662173088 on /run/cephadm/a2bb64be-e9d1-11ee-95cd-87774f69a715.lock 2024-03-24T11:28:49.889 INFO:teuthology.orchestra.run.smithi145.stderr:Lock 139785662173088 acquired on /run/cephadm/a2bb64be-e9d1-11ee-95cd-87774f69a715.lock 2024-03-24T11:28:49.889 INFO:teuthology.orchestra.run.smithi145.stderr:Verifying IP 172.21.15.145 port 3300 ... 2024-03-24T11:28:49.890 INFO:teuthology.orchestra.run.smithi145.stderr:Verifying IP 172.21.15.145 port 6789 ... 2024-03-24T11:28:49.890 INFO:teuthology.orchestra.run.smithi145.stderr:Base mon IP is 172.21.15.145, final addrv is [v2:172.21.15.145:3300,v1:172.21.15.145:6789] 2024-03-24T11:28:49.892 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.145 metric 100 2024-03-24T11:28:49.892 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.145 metric 100 2024-03-24T11:28:49.895 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:28:49.895 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T11:28:49.895 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1787sec hoplimit 64 pref medium 2024-03-24T11:28:49.897 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:28:49.897 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-24T11:28:49.897 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-24T11:28:49.898 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T11:28:49.898 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7791/64 scope link 2024-03-24T11:28:49.898 INFO:teuthology.orchestra.run.smithi145.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-24T11:28:49.898 INFO:teuthology.orchestra.run.smithi145.stderr:Mon IP `172.21.15.145` is in CIDR network `172.21.0.0/20` 2024-03-24T11:28:49.898 INFO:teuthology.orchestra.run.smithi145.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:28:49.899 INFO:teuthology.orchestra.run.smithi145.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:28:50.000 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-24T11:28:50.560 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Getting image source signatures 2024-03-24T11:28:50.560 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-24T11:28:50.560 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-24T11:28:50.560 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-24T11:28:50.560 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-24T11:28:50.560 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-24T11:28:50.560 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-24T11:28:50.688 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: Writing manifest to image destination 2024-03-24T11:28:50.791 INFO:teuthology.orchestra.run.smithi145.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-24T11:28:51.167 INFO:teuthology.orchestra.run.smithi145.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-24T11:28:51.681 INFO:teuthology.orchestra.run.smithi145.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-24T11:28:51.681 INFO:teuthology.orchestra.run.smithi145.stderr:Extracting ceph user uid/gid from container image... 2024-03-24T11:28:52.080 INFO:teuthology.orchestra.run.smithi145.stderr:stat: 167 167 2024-03-24T11:28:52.619 INFO:teuthology.orchestra.run.smithi145.stderr:Creating initial keys... 2024-03-24T11:28:52.994 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph-authtool: AQB0DgBmxBcgNhAA8OIzANGeIh1ZvB79hB2Hfw== 2024-03-24T11:28:53.982 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph-authtool: AQB1DgBmfQ93NRAA66/zodApOJsXLTlMFDpBjw== 2024-03-24T11:28:54.812 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph-authtool: AQB2DgBmAt7uKxAA9sbfau6w3itK4hEF6JqRMA== 2024-03-24T11:28:55.318 INFO:teuthology.orchestra.run.smithi145.stderr:Creating initial monmap... 2024-03-24T11:28:55.667 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:28:55.667 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-24T11:28:55.668 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:28:55.668 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:28:56.132 INFO:teuthology.orchestra.run.smithi145.stderr:monmaptool for a [v2:172.21.15.145:3300,v1:172.21.15.145:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:28:56.132 INFO:teuthology.orchestra.run.smithi145.stderr:setting min_mon_release = octopus 2024-03-24T11:28:56.132 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/monmaptool: set fsid to a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:28:56.132 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:28:56.132 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:28:56.132 INFO:teuthology.orchestra.run.smithi145.stderr:Creating mon... 2024-03-24T11:28:57.071 INFO:teuthology.orchestra.run.smithi145.stderr:create mon.a on 2024-03-24T11:28:57.277 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:28:57.420 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715.target. 2024-03-24T11:28:57.420 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715.target. 2024-03-24T11:28:57.717 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Failed to reset failed state of unit ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T11:28:57.724 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715.target.wants/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@.service. 2024-03-24T11:28:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:58 smithi145 podman[24484]: 2024-03-24 11:28:58.307441176 +0000 UTC m=+0.313418602 container start 8a1fe4e793682ffe1b0cfc9f387d6b6536e4752f653b958799b41439ef44b5bb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, architecture=x86_64, name=centos-stream, vendor=Red Hat, Inc., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, release=754, vcs-type=git, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.openshift.expose-services=) 2024-03-24T11:28:58.408 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: disabled 2024-03-24T11:28:58.415 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: inactive 2024-03-24T11:28:58.416 INFO:teuthology.orchestra.run.smithi145.stderr:firewalld.service is not enabled 2024-03-24T11:28:58.416 INFO:teuthology.orchestra.run.smithi145.stderr:Not possible to enable service . firewalld.service is not available 2024-03-24T11:28:58.416 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for mon to start... 2024-03-24T11:28:58.416 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for mon... 2024-03-24T11:28:58.631 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:58 smithi145 ceph-mon[24503]: mkfs a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:28:58.632 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:58 smithi145 ceph-mon[24503]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:28:58.632 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:58 smithi145 bash[24484]: 8a1fe4e793682ffe1b0cfc9f387d6b6536e4752f653b958799b41439ef44b5bb 2024-03-24T11:28:58.632 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:58 smithi145 systemd[1]: Started Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:28:58.903 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: cluster: 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: id: a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: services: 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.533445s) 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: data: 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: pgs: 2024-03-24T11:28:58.904 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:28:59.476 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:59 smithi145 ceph-mon[24503]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:28:59.476 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:59 smithi145 ceph-mon[24503]: monmap e1: 1 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} 2024-03-24T11:28:59.476 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:59 smithi145 ceph-mon[24503]: fsmap 2024-03-24T11:28:59.476 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:59 smithi145 ceph-mon[24503]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:28:59.477 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:59 smithi145 ceph-mon[24503]: mgrmap e1: no daemons active 2024-03-24T11:28:59.477 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:28:59 smithi145 ceph-mon[24503]: from='client.? 172.21.15.145:0/696934336' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T11:28:59.479 INFO:teuthology.orchestra.run.smithi145.stderr:mon is available 2024-03-24T11:28:59.479 INFO:teuthology.orchestra.run.smithi145.stderr:Assimilating anything we can from ceph.conf... 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: [global] 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: fsid = a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.145:3300,v1:172.21.15.145:6789] 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-24T11:28:59.971 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:28:59.972 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: [mgr] 2024-03-24T11:28:59.972 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-24T11:28:59.972 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-24T11:28:59.972 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:28:59.972 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: [osd] 2024-03-24T11:28:59.972 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-24T11:28:59.972 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-24T11:29:00.582 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:00 smithi145 ceph-mon[24503]: from='client.? 172.21.15.145:0/3844779604' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:29:00.583 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:00 smithi145 ceph-mon[24503]: from='client.? 172.21.15.145:0/3844779604' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:29:00.585 INFO:teuthology.orchestra.run.smithi145.stderr:Generating new minimal ceph.conf... 2024-03-24T11:29:01.474 INFO:teuthology.orchestra.run.smithi145.stderr:Restarting the monitor... 2024-03-24T11:29:01.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:01 smithi145 ceph-mon[24503]: from='client.? 172.21.15.145:0/228844090' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:29:01.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:01 smithi145 systemd[1]: Stopping Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:29:01.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:01 smithi145 bash[24721]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-24T11:29:02.092 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:01 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a[24499]: 2024-03-24T11:29:01.632+0000 7f2834448700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:29:02.092 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:01 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a[24499]: 2024-03-24T11:29:01.632+0000 7f2834448700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:29:02.092 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:01 smithi145 podman[24734]: 2024-03-24 11:29:01.868791999 +0000 UTC m=+0.343814433 container died 8a1fe4e793682ffe1b0cfc9f387d6b6536e4752f653b958799b41439ef44b5bb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, release=754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, version=8, com.redhat.component=centos-stream-container) 2024-03-24T11:29:02.344 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 podman[24734]: 2024-03-24 11:29:02.174349073 +0000 UTC m=+0.649371516 container cleanup 8a1fe4e793682ffe1b0cfc9f387d6b6536e4752f653b958799b41439ef44b5bb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-type=git, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8) 2024-03-24T11:29:02.344 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 bash[24734]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:29:02.344 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 bash[24762]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-24T11:29:02.344 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 podman[24749]: 2024-03-24 11:29:02.341769401 +0000 UTC m=+0.471685367 container remove 8a1fe4e793682ffe1b0cfc9f387d6b6536e4752f653b958799b41439ef44b5bb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64) 2024-03-24T11:29:02.344 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T11:29:02.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 systemd[1]: Stopped Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:29:02.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 systemd[1]: Starting Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:29:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 podman[24834]: 2024-03-24 11:29:02.542721912 +0000 UTC m=+0.090544354 container create ca31a24577e69c1ac2ba3d38b4326a941d78f341dd9b95e6b414df2973548680 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, name=centos-stream, vendor=Red Hat, Inc., distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD) 2024-03-24T11:29:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 podman[24834]: 2024-03-24 11:29:02.470562394 +0000 UTC m=+0.018384855 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-24T11:29:02.815 INFO:teuthology.orchestra.run.smithi145.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 podman[24834]: 2024-03-24 11:29:02.736730566 +0000 UTC m=+0.284553017 container init ca31a24577e69c1ac2ba3d38b4326a941d78f341dd9b95e6b414df2973548680 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.19.8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_CLEAN=True, name=centos-stream, GIT_BRANCH=HEAD) 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 podman[24834]: 2024-03-24 11:29:02.73992812 +0000 UTC m=+0.287750554 container start ca31a24577e69c1ac2ba3d38b4326a941d78f341dd9b95e6b414df2973548680 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, maintainer=Guillaume Abrioux , ceph=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: pidfile_write: ignore empty --pid-file 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: load: jerasure load: lrc 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: RocksDB version: 6.15.5 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Compile date Apr 18 2022 2024-03-24T11:29:02.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: DB SUMMARY 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: DB Session ID: RBSM6DV9A6ZIR6JEJ08X 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82657 ; 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.env: 0x5560e6ac6860 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.fs: Posix File System 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.info_log: 0x5560e8d5bdc0 2024-03-24T11:29:02.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.statistics: (nil) 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.use_fsync: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:29:02.941 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.db_log_dir: 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:29:02.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:29:02.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:29:02.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.write_buffer_manager: 0x5560e8e4c240 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.unordered_write: 0 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:29:02.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.row_cache: None 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.wal_filter: None 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.preserve_deletes: 0 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:29:02.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:29:02.946 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_open_files: -1 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:29:02.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Compression algorithms supported: 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kZSTD supported: 0 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.merge_operator: 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_filter: None 2024-03-24T11:29:02.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5560e8d27d00) 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_type: 0 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: data_block_index_type: 0 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_shortening: 1 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: hash_index_allow_collision: 1 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: checksum: 1 2024-03-24T11:29:02.949 INFO:journalctl@ceph.mon.a.smithi145.stdout: no_block_cache: 0 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache: 0x5560e8d92170 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_options: 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: capacity : 536870912 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: num_shard_bits : 4 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: strict_capacity_limit : 0 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_compressed: (nil) 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: persistent_cache: (nil) 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_size: 4096 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_size_deviation: 10 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_restart_interval: 16 2024-03-24T11:29:02.950 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_block_restart_interval: 1 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: metadata_block_size: 4096 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: partition_filters: 0 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: use_delta_encoding: 1 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: whole_key_filtering: 1 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: verify_compression: 0 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: format_version: 4 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: enable_index_compression: 1 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_align: 0 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression: NoCompression 2024-03-24T11:29:02.951 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.num_levels: 7 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:29:02.952 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:29:02.953 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.arena_block_size: 4194304 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-24T11:29:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:29:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:29:02.956 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.ttl: 2592000 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279742766778, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-24T11:29:02.957 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279742767496, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79207, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77470, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71779, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711279742, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "d2c68081-9ae5-40d1-9e17-1c2876f6a43d", "db_session_id": "RBSM6DV9A6ZIR6JEJ08X"}} 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279742767838, "job": 1, "event": "recovery_finished"} 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5560e8d78a80 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: DB pointer 0x5560e8d88000 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** DB Stats ** 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:29:02.958 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.000 0 0 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.000 0 0 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.000 0 0 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.0 0.00 0.00 1 0.000 0 0 2024-03-24T11:29:02.959 INFO:journalctl@ceph.mon.a.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative compaction: 0.00 GB write, 25.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval compaction: 0.00 GB write, 25.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:29:02.960 INFO:journalctl@ceph.mon.a.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.000 0 0 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.000 0 0 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.0 0.00 0.00 1 0.000 0 0 2024-03-24T11:29:02.961 INFO:journalctl@ceph.mon.a.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative compaction: 0.00 GB write, 25.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: starting mon.a rank 0 at public addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] at bind addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???) e1 preinit fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???).mds e1 new map 2024-03-24T11:29:02.962 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???).mds e1 print_map 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout: e1 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout: legacy client fscid: -1 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout: No filesystems configured 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expand map: {default=false} 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta from 'false' to 'false' 2024-03-24T11:29:02.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expanded map: {default=false} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expand map: {default=info} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta from 'info' to 'info' 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expanded map: {default=info} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expand map: {default=daemon} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta from 'daemon' to 'daemon' 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expanded map: {default=daemon} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expand map: {default=debug} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta from 'debug' to 'debug' 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: expand_channel_meta expanded map: {default=debug} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 bash[24834]: ca31a24577e69c1ac2ba3d38b4326a941d78f341dd9b95e6b414df2973548680 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 systemd[1]: Started Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: monmap e1: 1 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: fsmap 2024-03-24T11:29:02.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:29:02.965 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:02 smithi145 ceph-mon[24853]: mgrmap e1: no daemons active 2024-03-24T11:29:03.870 INFO:teuthology.orchestra.run.smithi145.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:29:03.870 INFO:teuthology.orchestra.run.smithi145.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:29:03.870 INFO:teuthology.orchestra.run.smithi145.stderr:Creating mgr... 2024-03-24T11:29:03.870 INFO:teuthology.orchestra.run.smithi145.stderr:Verifying port 9283 ... 2024-03-24T11:29:04.039 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Failed to reset failed state of unit ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T11:29:04.046 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715.target.wants/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@.service. 2024-03-24T11:29:04.498 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:04 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1746676364' entity='client.admin' 2024-03-24T11:29:04.717 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: disabled 2024-03-24T11:29:04.726 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: inactive 2024-03-24T11:29:04.726 INFO:teuthology.orchestra.run.smithi145.stderr:firewalld.service is not enabled 2024-03-24T11:29:04.726 INFO:teuthology.orchestra.run.smithi145.stderr:Not possible to enable service . firewalld.service is not available 2024-03-24T11:29:04.733 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: disabled 2024-03-24T11:29:04.740 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: inactive 2024-03-24T11:29:04.741 INFO:teuthology.orchestra.run.smithi145.stderr:firewalld.service is not enabled 2024-03-24T11:29:04.741 INFO:teuthology.orchestra.run.smithi145.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-24T11:29:04.742 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for mgr to start... 2024-03-24T11:29:04.742 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for mgr... 2024-03-24T11:29:04.854 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:04 smithi145 systemd[1]: Started Ceph mgr.y for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:29:05.238 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: { 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "fsid": "a2bb64be-e9d1-11ee-95cd-87774f69a715", 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "health": { 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 0 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:05.239 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "a" 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T11:29:05.240 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:05.244 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "available": false, 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "iostat", 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "nfs", 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "restful" 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T11:29:05.245 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:05.246 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "modified": "2024-03-24T11:28:58.369847+0000", 2024-03-24T11:29:05.246 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:29:05.246 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:05.246 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T11:29:05.246 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: } 2024-03-24T11:29:05.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:05.108+0000 7f676ca36000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:29:05.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:05.202+0000 7f676ca36000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:29:05.717 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:05 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3527054459' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:29:05.720 INFO:teuthology.orchestra.run.smithi145.stderr:mgr not available, waiting (1/15)... 2024-03-24T11:29:05.981 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:05.810+0000 7f676ca36000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:29:05.981 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:05.898+0000 7f676ca36000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:29:05.981 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:05.980+0000 7f676ca36000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:29:06.314 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:06.128+0000 7f676ca36000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:29:06.314 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:06.207+0000 7f676ca36000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:29:06.315 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:06.314+0000 7f676ca36000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:29:06.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:06.396+0000 7f676ca36000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:29:06.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:06.557+0000 7f676ca36000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:29:07.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:06.642+0000 7f676ca36000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:29:07.935 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:07.663+0000 7f676ca36000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:29:07.935 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:07.803+0000 7f676ca36000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:29:08.204 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:07.997+0000 7f676ca36000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:29:08.204 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:08.087+0000 7f676ca36000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:29:08.210 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:29:08.210 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: { 2024-03-24T11:29:08.210 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "fsid": "a2bb64be-e9d1-11ee-95cd-87774f69a715", 2024-03-24T11:29:08.210 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "health": { 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 0 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "a" 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:08.211 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T11:29:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "available": false, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T11:29:08.213 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "iostat", 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "nfs", 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "restful" 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "modified": "2024-03-24T11:28:58.369847+0000", 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T11:29:08.214 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: } 2024-03-24T11:29:08.555 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:08.231+0000 7f676ca36000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:29:08.555 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:08.554+0000 7f676ca36000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:29:08.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:08 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1629147638' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:29:08.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:08.777+0000 7f676ca36000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:29:09.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:08.861+0000 7f676ca36000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:29:09.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:09.030+0000 7f676ca36000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:29:10.251 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:09.956+0000 7f676ca36000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:29:10.251 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:10.251+0000 7f676ca36000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:29:10.562 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:10.346+0000 7f676ca36000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:29:10.562 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:10.496+0000 7f676ca36000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:29:10.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: Activating manager daemon y 2024-03-24T11:29:10.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: mgrmap e2: y(active, starting, since 0.00283321s) 2024-03-24T11:29:10.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:29:10.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:29:10.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: Manager daemon y is now available 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' 2024-03-24T11:29:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:10 smithi145 ceph-mon[24853]: from='mgr.14100 172.21.15.145:0/4189296911' entity='mgr.y' 2024-03-24T11:29:12.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:12 smithi145 ceph-mon[24853]: mgrmap e3: y(active, since 1.00653s) 2024-03-24T11:29:12.781 INFO:teuthology.orchestra.run.smithi145.stderr:mgr not available, waiting (2/15)... 2024-03-24T11:29:14.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:14 smithi145 ceph-mon[24853]: mgrmap e4: y(active, since 3s) 2024-03-24T11:29:18.853 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: { 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "fsid": "a2bb64be-e9d1-11ee-95cd-87774f69a715", 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "health": { 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "checks": {}, 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mutes": [] 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum": [ 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 0 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:18.854 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "a" 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "quorum_age": 15, 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "monmap": { 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osdmap": { 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:18.855 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-24T11:29:18.856 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-24T11:29:18.856 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-24T11:29:18.856 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-24T11:29:18.859 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-24T11:29:18.859 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "pgmap": { 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "fsmap": { 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-24T11:29:18.860 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "available": true, 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "modules": [ 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "iostat", 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "nfs", 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "restful" 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ], 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "servicemap": { 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "modified": "2024-03-24T11:28:58.369847+0000", 2024-03-24T11:29:18.861 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "services": {} 2024-03-24T11:29:18.862 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: }, 2024-03-24T11:29:18.862 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-24T11:29:18.862 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: } 2024-03-24T11:29:19.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:18 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3303658791' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:29:20.279 INFO:teuthology.orchestra.run.smithi145.stderr:mgr is available 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: [global] 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: fsid = a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-24T11:29:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:29:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: [mgr] 2024-03-24T11:29:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-24T11:29:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 2024-03-24T11:29:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: [osd] 2024-03-24T11:29:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-24T11:29:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-24T11:29:21.199 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:20 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2729826022' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:29:21.199 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:20 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2729826022' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:29:21.469 INFO:teuthology.orchestra.run.smithi145.stderr:Enabling cephadm module... 2024-03-24T11:29:22.730 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:22 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4073226158' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:29:22.730 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:22 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setuser ceph since I am not root 2024-03-24T11:29:22.731 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:22 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setgroup ceph since I am not root 2024-03-24T11:29:22.998 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:22 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:22.907+0000 7fda481ff000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:29:22.998 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:22 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:22.997+0000 7fda481ff000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:29:23.605 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:23 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4073226158' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:29:23.606 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:23 smithi145 ceph-mon[24853]: mgrmap e5: y(active, since 11s) 2024-03-24T11:29:23.637 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: { 2024-03-24T11:29:23.638 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-24T11:29:23.638 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "available": true, 2024-03-24T11:29:23.638 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-24T11:29:23.638 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-24T11:29:23.638 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: } 2024-03-24T11:29:23.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:23.605+0000 7fda481ff000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:29:23.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:23.698+0000 7fda481ff000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:29:23.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:23.781+0000 7fda481ff000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:29:24.204 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:23.931+0000 7fda481ff000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:29:24.204 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:24.012+0000 7fda481ff000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:29:24.204 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:24.120+0000 7fda481ff000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:29:24.205 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:24.203+0000 7fda481ff000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:29:24.230 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for the mgr to restart... 2024-03-24T11:29:24.231 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for mgr epoch 5... 2024-03-24T11:29:24.459 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:24 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2407509666' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:29:24.459 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:24.372+0000 7fda481ff000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:29:24.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:24.458+0000 7fda481ff000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:29:25.795 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:25.478+0000 7fda481ff000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:29:25.795 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:25.606+0000 7fda481ff000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:29:25.796 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:25.795+0000 7fda481ff000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:29:26.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:25.882+0000 7fda481ff000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:29:26.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:26 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:26.017+0000 7fda481ff000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:29:26.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:26 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:26.335+0000 7fda481ff000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:29:26.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:26 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:26.555+0000 7fda481ff000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:29:27.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:26 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:26.638+0000 7fda481ff000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:29:27.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:26 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:26.806+0000 7fda481ff000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:29:28.010 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:27 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:27.719+0000 7fda481ff000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:29:28.010 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:28 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:28.009+0000 7fda481ff000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:29:28.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:28 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:28.104+0000 7fda481ff000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:29:28.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:28 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:28.254+0000 7fda481ff000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:29:29.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:28 smithi145 ceph-mon[24853]: Active manager daemon y restarted 2024-03-24T11:29:29.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:28 smithi145 ceph-mon[24853]: Activating manager daemon y 2024-03-24T11:29:29.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:28 smithi145 ceph-mon[24853]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:29:29.791 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: { 2024-03-24T11:29:29.792 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-24T11:29:29.792 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "initialized": true 2024-03-24T11:29:29.792 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: } 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: mgrmap e6: y(active, starting, since 0.511854s) 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: Manager daemon y is now available 2024-03-24T11:29:30.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:30.085 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:30.085 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:30.085 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:30.085 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:30.085 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:29:30.085 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:29 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:29:30.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:29:30] ENGINE Bus STARTING 2024-03-24T11:29:30.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:29:30] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:29:30.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:29:30] ENGINE Bus STARTED 2024-03-24T11:29:30.370 INFO:teuthology.orchestra.run.smithi145.stderr:mgr epoch 5 is available 2024-03-24T11:29:30.370 INFO:teuthology.orchestra.run.smithi145.stderr:Setting orchestrator backend to cephadm... 2024-03-24T11:29:30.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-mon[24853]: mgrmap e7: y(active, since 1.51637s) 2024-03-24T11:29:30.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-mon[24853]: [24/Mar/2024:11:29:30] ENGINE Bus STARTING 2024-03-24T11:29:30.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-mon[24853]: [24/Mar/2024:11:29:30] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:29:30.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-mon[24853]: [24/Mar/2024:11:29:30] ENGINE Bus STARTED 2024-03-24T11:29:30.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:30.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:30 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:31 smithi145 ceph-mon[24853]: mgrmap e8: y(active, since 2s) 2024-03-24T11:29:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:31 smithi145 ceph-mon[24853]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:31 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:31 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:32.040 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: value unchanged 2024-03-24T11:29:32.607 INFO:teuthology.orchestra.run.smithi145.stderr:Generating ssh key... 2024-03-24T11:29:33.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:32 smithi145 ceph-mon[24853]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-mon[24853]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-mon[24853]: Generating ssh key... 2024-03-24T11:29:34.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: Generating public/private rsa key pair. 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: Your identification has been saved in /tmp/tmpdatyawgd/key. 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: Your public key has been saved in /tmp/tmpdatyawgd/key.pub. 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: The key fingerprint is: 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: SHA256:x9DvNgHsDk1LDzO0qpkc0fN0LFYPt0wE1Jp9LF7iZxY ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: The key's randomart image is: 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: +---[RSA 3072]----+ 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | ..=++ | 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | . + + *.. | 2024-03-24T11:29:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | . + # o++. | 2024-03-24T11:29:34.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | . & @o +E+| 2024-03-24T11:29:34.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | . S B +o =.| 2024-03-24T11:29:34.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | . = + . .o +| 2024-03-24T11:29:34.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | = . + + | 2024-03-24T11:29:34.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | . . | 2024-03-24T11:29:34.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: | | 2024-03-24T11:29:34.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: +----[SHA256]-----+ 2024-03-24T11:29:35.109 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRI6FAN+t0jCyEQOG/IbuUwQ3Hw0Amrcnikx8KYXwyLEmBtrNmaQ7gsAyNo9srr+PhxzImDMetKU+elCTv9rbHZzEz2phPgS0S5NsMvK/l8yzXgP4v6vhsT7z/Hr2Q6aueXFPBUXM9bNpGQ+PUoXIRMTRrcTm+mfJpvCCQbSCwnprFUJVgNxRyBo6WrvQjTkHKTszTJCJmC0etvL/n88Ztfbx+6npDQIlYJjqa0UEgpCbZD/tkLvdSXmmxpKuZfFNJ9NNjHC1Md0PuKRAjEN4P4xOqC2F97655DW0JgVN+S9zXcNy7GZEkTlr6XLfeB45r+oPrCZp89/lwdtwwQ0fP47xlrvD7fziK/UxHMQ/8W5zDvl0k8bOO3pb/gUVea8IHfVTxuv/ipRMpYjmcO8z644FqngAxxa7KSiG5lQP3czi4NAZ6rfoLKQv85K0ITYjChTbMsqmrvcyMVF/nRJWORuXbtAxouYTeYy+p9sTZcV2JnBM9TFurtnZusH4GNxs= ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:29:35.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:34 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:35.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:34 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:35.628 INFO:teuthology.orchestra.run.smithi145.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:29:35.628 INFO:teuthology.orchestra.run.smithi145.stderr:Adding key to root@localhost authorized_keys... 2024-03-24T11:29:35.628 INFO:teuthology.orchestra.run.smithi145.stderr:Adding host smithi145... 2024-03-24T11:29:36.248 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:35 smithi145 ceph-mon[24853]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:37.234 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:36 smithi145 ceph-mon[24853]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi145", "addr": "172.21.15.145", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:37.241 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: Added host 'smithi145' with addr '172.21.15.145' 2024-03-24T11:29:37.859 INFO:teuthology.orchestra.run.smithi145.stderr:Deploying unmanaged mon service... 2024-03-24T11:29:38.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:38 smithi145 ceph-mon[24853]: Deploying cephadm binary to smithi145 2024-03-24T11:29:38.512 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:38 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:38.512 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:38 smithi145 ceph-mon[24853]: Added host smithi145 2024-03-24T11:29:38.512 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:38 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:38.512 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-24T11:29:39.294 INFO:teuthology.orchestra.run.smithi145.stderr:Deploying unmanaged mgr service... 2024-03-24T11:29:39.601 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:39 smithi145 ceph-mon[24853]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:39.602 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:39 smithi145 ceph-mon[24853]: Saving service mon spec with placement count:5 2024-03-24T11:29:39.602 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:39 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:39.911 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-24T11:29:41.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:40 smithi145 ceph-mon[24853]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:41.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:40 smithi145 ceph-mon[24853]: Saving service mgr spec with placement count:2 2024-03-24T11:29:41.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:40 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:42.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:42 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:42.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:42 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:42.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:42 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/274455189' entity='client.admin' 2024-03-24T11:29:43.702 INFO:teuthology.orchestra.run.smithi145.stderr:Enabling the dashboard module... 2024-03-24T11:29:44.011 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:43 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2973732926' entity='client.admin' 2024-03-24T11:29:44.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:44 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4180325177' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:29:44.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:44 smithi145 ceph-mon[24853]: from='mgr.14118 172.21.15.145:0/2856552700' entity='mgr.y' 2024-03-24T11:29:45.283 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setuser ceph since I am not root 2024-03-24T11:29:45.284 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setgroup ceph since I am not root 2024-03-24T11:29:45.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:45.403+0000 7fe7caf62000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:29:45.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:45.491+0000 7fe7caf62000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:29:46.187 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:45 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4180325177' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:29:46.187 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:45 smithi145 ceph-mon[24853]: mgrmap e9: y(active, since 16s) 2024-03-24T11:29:46.187 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.100+0000 7fe7caf62000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:29:46.188 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: { 2024-03-24T11:29:46.188 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-24T11:29:46.188 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "available": true, 2024-03-24T11:29:46.188 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-24T11:29:46.188 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-24T11:29:46.188 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: } 2024-03-24T11:29:46.509 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.191+0000 7fe7caf62000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:29:46.510 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.279+0000 7fe7caf62000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:29:46.510 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.429+0000 7fe7caf62000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:29:46.510 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.509+0000 7fe7caf62000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:29:46.717 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for the mgr to restart... 2024-03-24T11:29:46.717 INFO:teuthology.orchestra.run.smithi145.stderr:Waiting for mgr epoch 9... 2024-03-24T11:29:46.840 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.619+0000 7fe7caf62000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:29:46.840 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.703+0000 7fe7caf62000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:29:47.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2927218452' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:29:47.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.873+0000 7fe7caf62000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:29:47.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:46.960+0000 7fe7caf62000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:29:48.303 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:47.983+0000 7fe7caf62000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:29:48.303 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:48 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:48.112+0000 7fe7caf62000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:29:48.303 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:48 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:48.302+0000 7fe7caf62000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:29:48.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:48 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:48.390+0000 7fe7caf62000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:29:48.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:48 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:48.525+0000 7fe7caf62000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:29:49.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:48 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:48.846+0000 7fe7caf62000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:29:49.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:49.067+0000 7fe7caf62000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:29:49.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:49.151+0000 7fe7caf62000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:29:49.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:49.322+0000 7fe7caf62000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:29:50.513 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:50.233+0000 7fe7caf62000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:29:50.771 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:50.525+0000 7fe7caf62000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:29:50.771 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:50.620+0000 7fe7caf62000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:29:50.771 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:29:50.770+0000 7fe7caf62000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:29:51.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:50 smithi145 ceph-mon[24853]: Active manager daemon y restarted 2024-03-24T11:29:51.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:50 smithi145 ceph-mon[24853]: Activating manager daemon y 2024-03-24T11:29:51.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:50 smithi145 ceph-mon[24853]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:29:51.304 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:29:51] ENGINE Bus STARTING 2024-03-24T11:29:51.597 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:29:51] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:29:51.598 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:29:51] ENGINE Bus STARTED 2024-03-24T11:29:51.852 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: { 2024-03-24T11:29:51.852 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-24T11:29:51.852 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: "initialized": true 2024-03-24T11:29:51.852 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: } 2024-03-24T11:29:52.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: mgrmap e10: y(active, starting, since 0.0523761s) 2024-03-24T11:29:52.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:29:52.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:29:52.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:29:52.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:29:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:29:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: Manager daemon y is now available 2024-03-24T11:29:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:29:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:29:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:29:52.459 INFO:teuthology.orchestra.run.smithi145.stderr:mgr epoch 9 is available 2024-03-24T11:29:52.460 INFO:teuthology.orchestra.run.smithi145.stderr:Generating a dashboard self-signed certificate... 2024-03-24T11:29:53.107 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:52 smithi145 ceph-mon[24853]: [24/Mar/2024:11:29:51] ENGINE Bus STARTING 2024-03-24T11:29:53.107 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:52 smithi145 ceph-mon[24853]: [24/Mar/2024:11:29:51] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:29:53.107 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:52 smithi145 ceph-mon[24853]: [24/Mar/2024:11:29:51] ENGINE Bus STARTED 2024-03-24T11:29:53.107 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:52 smithi145 ceph-mon[24853]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:29:53.107 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:52 smithi145 ceph-mon[24853]: mgrmap e11: y(active, since 1.06104s) 2024-03-24T11:29:53.107 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:52 smithi145 ceph-mon[24853]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:29:53.108 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-24T11:29:53.649 INFO:teuthology.orchestra.run.smithi145.stderr:Creating initial admin user... 2024-03-24T11:29:54.259 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:29:54.259 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:29:54.259 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:29:54.259 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:29:54.259 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:53 smithi145 ceph-mon[24853]: mgrmap e12: y(active, since 2s) 2024-03-24T11:29:54.526 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$p5cn/Jjxz0IQko9/DR/Aa.Stm/RTKzwi1w6BmIgiut3DthIUZ5npm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711279794, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:29:55.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:54 smithi145 ceph-mon[24853]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:55.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:54 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:29:55.276 INFO:teuthology.orchestra.run.smithi145.stderr:Fetching dashboard port number... 2024-03-24T11:29:56.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:55 smithi145 ceph-mon[24853]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:29:56.372 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: 8443 2024-03-24T11:29:57.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:56 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1716410208' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:29:57.053 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: disabled 2024-03-24T11:29:57.060 INFO:teuthology.orchestra.run.smithi145.stderr:systemctl: inactive 2024-03-24T11:29:57.061 INFO:teuthology.orchestra.run.smithi145.stderr:firewalld.service is not enabled 2024-03-24T11:29:57.061 INFO:teuthology.orchestra.run.smithi145.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:29:57.063 INFO:teuthology.orchestra.run.smithi145.stderr:Ceph Dashboard is now available at: 2024-03-24T11:29:57.063 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:57.063 INFO:teuthology.orchestra.run.smithi145.stderr: URL: https://smithi145.front.sepia.ceph.com:8443/ 2024-03-24T11:29:57.063 INFO:teuthology.orchestra.run.smithi145.stderr: User: admin 2024-03-24T11:29:57.063 INFO:teuthology.orchestra.run.smithi145.stderr: Password: pvdqq3lmoh 2024-03-24T11:29:57.063 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:57.064 INFO:teuthology.orchestra.run.smithi145.stderr:Enabling autotune for osd_memory_target 2024-03-24T11:29:58.239 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:57 smithi145 ceph-mon[24853]: mgrmap e13: y(active, since 6s) 2024-03-24T11:29:58.932 INFO:teuthology.orchestra.run.smithi145.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr:You can access the Ceph CLI with: 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr: ceph telemetry on 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr:For more information see: 2024-03-24T11:29:59.448 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:59.449 INFO:teuthology.orchestra.run.smithi145.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-24T11:29:59.449 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:29:59.449 INFO:teuthology.orchestra.run.smithi145.stderr:Bootstrap complete. 2024-03-24T11:29:59.488 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:29:59.489 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:29:59.489 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:29:59.504 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:29:59.505 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:29:59.505 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:29:59.560 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:29:59.560 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:29:59.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T11:29:59.628 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:29:59.628 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:29:59.628 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:29:59.683 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:29:59.683 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRI6FAN+t0jCyEQOG/IbuUwQ3Hw0Amrcnikx8KYXwyLEmBtrNmaQ7gsAyNo9srr+PhxzImDMetKU+elCTv9rbHZzEz2phPgS0S5NsMvK/l8yzXgP4v6vhsT7z/Hr2Q6aueXFPBUXM9bNpGQ+PUoXIRMTRrcTm+mfJpvCCQbSCwnprFUJVgNxRyBo6WrvQjTkHKTszTJCJmC0etvL/n88Ztfbx+6npDQIlYJjqa0UEgpCbZD/tkLvdSXmmxpKuZfFNJ9NNjHC1Md0PuKRAjEN4P4xOqC2F97655DW0JgVN+S9zXcNy7GZEkTlr6XLfeB45r+oPrCZp89/lwdtwwQ0fP47xlrvD7fziK/UxHMQ/8W5zDvl0k8bOO3pb/gUVea8IHfVTxuv/ipRMpYjmcO8z644FqngAxxa7KSiG5lQP3czi4NAZ6rfoLKQv85K0ITYjChTbMsqmrvcyMVF/nRJWORuXbtAxouYTeYy+p9sTZcV2JnBM9TFurtnZusH4GNxs= ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:29:59.762 INFO:teuthology.orchestra.run.smithi145.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRI6FAN+t0jCyEQOG/IbuUwQ3Hw0Amrcnikx8KYXwyLEmBtrNmaQ7gsAyNo9srr+PhxzImDMetKU+elCTv9rbHZzEz2phPgS0S5NsMvK/l8yzXgP4v6vhsT7z/Hr2Q6aueXFPBUXM9bNpGQ+PUoXIRMTRrcTm+mfJpvCCQbSCwnprFUJVgNxRyBo6WrvQjTkHKTszTJCJmC0etvL/n88Ztfbx+6npDQIlYJjqa0UEgpCbZD/tkLvdSXmmxpKuZfFNJ9NNjHC1Md0PuKRAjEN4P4xOqC2F97655DW0JgVN+S9zXcNy7GZEkTlr6XLfeB45r+oPrCZp89/lwdtwwQ0fP47xlrvD7fziK/UxHMQ/8W5zDvl0k8bOO3pb/gUVea8IHfVTxuv/ipRMpYjmcO8z644FqngAxxa7KSiG5lQP3czi4NAZ6rfoLKQv85K0ITYjChTbMsqmrvcyMVF/nRJWORuXbtAxouYTeYy+p9sTZcV2JnBM9TFurtnZusH4GNxs= ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:29:59.776 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRI6FAN+t0jCyEQOG/IbuUwQ3Hw0Amrcnikx8KYXwyLEmBtrNmaQ7gsAyNo9srr+PhxzImDMetKU+elCTv9rbHZzEz2phPgS0S5NsMvK/l8yzXgP4v6vhsT7z/Hr2Q6aueXFPBUXM9bNpGQ+PUoXIRMTRrcTm+mfJpvCCQbSCwnprFUJVgNxRyBo6WrvQjTkHKTszTJCJmC0etvL/n88Ztfbx+6npDQIlYJjqa0UEgpCbZD/tkLvdSXmmxpKuZfFNJ9NNjHC1Md0PuKRAjEN4P4xOqC2F97655DW0JgVN+S9zXcNy7GZEkTlr6XLfeB45r+oPrCZp89/lwdtwwQ0fP47xlrvD7fziK/UxHMQ/8W5zDvl0k8bOO3pb/gUVea8IHfVTxuv/ipRMpYjmcO8z644FqngAxxa7KSiG5lQP3czi4NAZ6rfoLKQv85K0ITYjChTbMsqmrvcyMVF/nRJWORuXbtAxouYTeYy+p9sTZcV2JnBM9TFurtnZusH4GNxs= ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:29:59.822 INFO:teuthology.orchestra.run.smithi149.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRI6FAN+t0jCyEQOG/IbuUwQ3Hw0Amrcnikx8KYXwyLEmBtrNmaQ7gsAyNo9srr+PhxzImDMetKU+elCTv9rbHZzEz2phPgS0S5NsMvK/l8yzXgP4v6vhsT7z/Hr2Q6aueXFPBUXM9bNpGQ+PUoXIRMTRrcTm+mfJpvCCQbSCwnprFUJVgNxRyBo6WrvQjTkHKTszTJCJmC0etvL/n88Ztfbx+6npDQIlYJjqa0UEgpCbZD/tkLvdSXmmxpKuZfFNJ9NNjHC1Md0PuKRAjEN4P4xOqC2F97655DW0JgVN+S9zXcNy7GZEkTlr6XLfeB45r+oPrCZp89/lwdtwwQ0fP47xlrvD7fziK/UxHMQ/8W5zDvl0k8bOO3pb/gUVea8IHfVTxuv/ipRMpYjmcO8z644FqngAxxa7KSiG5lQP3czi4NAZ6rfoLKQv85K0ITYjChTbMsqmrvcyMVF/nRJWORuXbtAxouYTeYy+p9sTZcV2JnBM9TFurtnZusH4GNxs= ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:29:59.837 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:30:00.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:29:59 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3411520261' entity='client.admin' 2024-03-24T11:30:02.788 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:30:02.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:30:02.814 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:02.814 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi145", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:02.814 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:02.815 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:02.815 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:02.815 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3994928702' entity='client.admin' 2024-03-24T11:30:02.815 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:02.815 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:05.118 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi149 2024-03-24T11:30:05.118 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-24T11:30:05.118 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:30:05.137 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-24T11:30:05.137 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:30:05.194 INFO:tasks.cephadm:Adding host smithi149 to orchestrator... 2024-03-24T11:30:05.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch host add smithi149 2024-03-24T11:30:05.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:05.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:05.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:05.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:05.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:05.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:30:05.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:30:05.732 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:05.732 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:07.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:07 smithi145 ceph-mon[24853]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:07.896 INFO:teuthology.orchestra.run.smithi145.stdout:Added host 'smithi149' with addr '172.21.15.149' 2024-03-24T11:30:08.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: Deploying cephadm binary to smithi149 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: Added host smithi149 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:09.077 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:10.241 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:30:10.241 INFO:teuthology.orchestra.run.smithi145.stdout:[{"addr": "172.21.15.145", "hostname": "smithi145", "labels": [], "status": ""}, {"addr": "172.21.15.149", "hostname": "smithi149", "labels": [], "status": ""}] 2024-03-24T11:30:10.735 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:30:10.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:30:10.760 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:10 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:11.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:11 smithi145 ceph-mon[24853]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:30:11.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:11 smithi145 ceph-mon[24853]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:13.284 INFO:teuthology.orchestra.run.smithi145.stderr:adjusted tunables profile to default 2024-03-24T11:30:13.597 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:13 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:13.597 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:13 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:13.597 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:13 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:13.597 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:13 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:13.597 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:13 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4278478795' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:30:13.597 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:13 smithi145 ceph-mon[24853]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:13.867 INFO:tasks.cephadm:Adding mon.a on smithi145 2024-03-24T11:30:13.867 INFO:tasks.cephadm:Adding mon.c on smithi145 2024-03-24T11:30:13.868 INFO:tasks.cephadm:Adding mon.b on smithi149 2024-03-24T11:30:13.868 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi145:172.21.15.145=a;smithi145:[v2:172.21.15.145:3301,v1:172.21.15.145:6790]=c;smithi149:172.21.15.149=b' 2024-03-24T11:30:14.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:14 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4278478795' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:30:14.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:14 smithi145 ceph-mon[24853]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:30:15.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:15 smithi145 ceph-mon[24853]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:15.642 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled mon update... 2024-03-24T11:30:16.313 DEBUG:teuthology.orchestra.run.smithi145:mon.c> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T11:30:16.316 DEBUG:teuthology.orchestra.run.smithi149:mon.b> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T11:30:16.318 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:30:16.319 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi145:172.21.15.145=a;smithi145:[v2:172.21.15.145:3301,v1:172.21.15.145:6790]=c;smithi149:172.21.15.149=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: Saving service mon spec with placement smithi145:172.21.15.145=a;smithi145:[v2:172.21.15.145:3301,v1:172.21.15.145:6790]=c;smithi149:172.21.15.149=b;count:3 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:30:16.843 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:30:16.844 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:16.844 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:16.844 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:16.844 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:17.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:17 smithi145 ceph-mon[24853]: Deploying daemon mon.c on smithi145 2024-03-24T11:30:17.731 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:17 smithi145 ceph-mon[24853]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:18.036 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:30:18.036 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"a2bb64be-e9d1-11ee-95cd-87774f69a715","modified":"2024-03-24T11:28:55.622211Z","created":"2024-03-24T11:28:55.622211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:30:18.038 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-03-24T11:30:19.024 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:18 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/2799767656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:30:19.576 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:30:19.577 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:30:24.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: mon.a calling monitor election 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: mon.c calling monitor election 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:24.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: monmap e2: 2 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: fsmap 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: mgrmap e13: y(active, since 33s) 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: overall HEALTH_OK 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:24.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:24 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:24.624 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:30:24.624 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":2,"fsid":"a2bb64be-e9d1-11ee-95cd-87774f69a715","modified":"2024-03-24T11:30:19.195868Z","created":"2024-03-24T11:28:55.622211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3301","nonce":0},{"type":"v1","addr":"172.21.15.145:6790","nonce":0}]},"addr":"172.21.15.145:6790/0","public_addr":"172.21.15.145:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:30:24.626 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 2 2024-03-24T11:30:25.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[24853]: Deploying daemon mon.b on smithi149 2024-03-24T11:30:25.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/4196237642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:30:25.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[24853]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:25.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:25.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[29394]: Deploying daemon mon.b on smithi149 2024-03-24T11:30:25.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/4196237642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:30:25.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[29394]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:25.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:26.356 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:30:26.357 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:30:28.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:27 smithi145 ceph-mon[24853]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:28.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:27 smithi145 ceph-mon[29394]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:28.726 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:30:28.726 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":2,"fsid":"a2bb64be-e9d1-11ee-95cd-87774f69a715","modified":"2024-03-24T11:30:19.195868Z","created":"2024-03-24T11:28:55.622211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3301","nonce":0},{"type":"v1","addr":"172.21.15.145:6790","nonce":0}]},"addr":"172.21.15.145:6790/0","public_addr":"172.21.15.145:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:30:28.728 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 2 2024-03-24T11:30:29.010 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:28 smithi149 ceph-mon[25685]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-24T11:30:29.362 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:29 smithi149 bash[25627]: 91790c8c1af1c6defad1cd55743c3eaa201d94023ed3f15101a09245042dbb23 2024-03-24T11:30:29.362 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:29 smithi149 systemd[1]: Started Ceph mon.b for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:30:30.385 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:30:30.385 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: mon.a calling monitor election 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: mon.c calling monitor election 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: mon.b calling monitor election 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: mon.a calling monitor election 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:30:34.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: fsmap 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: mgrmap e13: y(active, since 43s) 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: overall HEALTH_OK 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:34.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: mon.a calling monitor election 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: mon.c calling monitor election 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: mon.b calling monitor election 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: mon.a calling monitor election 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: fsmap 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:30:34.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: mgrmap e13: y(active, since 43s) 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: overall HEALTH_OK 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: mon.a calling monitor election 2024-03-24T11:30:34.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: mon.c calling monitor election 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: mon.b calling monitor election 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:34.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: mon.a calling monitor election 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: fsmap 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: mgrmap e13: y(active, since 43s) 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: overall HEALTH_OK 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:34.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:35.144 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-24T11:30:35.145 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":3,"fsid":"a2bb64be-e9d1-11ee-95cd-87774f69a715","modified":"2024-03-24T11:30:28.807400Z","created":"2024-03-24T11:28:55.622211Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3301","nonce":0},{"type":"v1","addr":"172.21.15.145:6790","nonce":0}]},"addr":"172.21.15.145:6790/0","public_addr":"172.21.15.145:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-24T11:30:35.147 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 3 2024-03-24T11:30:35.753 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:30:35.753 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:30:35.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/4013348856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:30:35.779 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:35.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:35 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/4013348856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:30:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:30:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:30:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:36.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:35 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/4013348856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:30:36.978 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:36 smithi145 ceph-mon[24853]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:30:36.979 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:36 smithi145 ceph-mon[24853]: Reconfiguring daemon mon.a on smithi145 2024-03-24T11:30:36.979 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:36 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:36.979 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:36 smithi145 ceph-mon[29394]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:30:36.979 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:36 smithi145 ceph-mon[29394]: Reconfiguring daemon mon.a on smithi145 2024-03-24T11:30:36.979 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:36 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:36 smithi149 ceph-mon[25685]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:30:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:36 smithi149 ceph-mon[25685]: Reconfiguring daemon mon.a on smithi145 2024-03-24T11:30:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:36 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:37.945 INFO:teuthology.orchestra.run.smithi145.stdout:# minimal ceph.conf for a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:30:37.945 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-03-24T11:30:37.945 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:30:37.945 INFO:teuthology.orchestra.run.smithi145.stdout: mon_host = [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] 2024-03-24T11:30:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:37 smithi149 ceph-mon[25685]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:37 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:37 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:37 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:37 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:38.310 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[24853]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[29394]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:38.311 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:37 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:38.637 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:30:38.638 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:30:38.638 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:30:38.710 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:30:38.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:30:38.777 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-24T11:30:38.778 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:30:38.810 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-24T11:30:38.810 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:30:38.880 INFO:tasks.cephadm:Adding mgr.y on smithi145 2024-03-24T11:30:38.880 INFO:tasks.cephadm:Adding mgr.x on smithi149 2024-03-24T11:30:38.881 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi145=y;smithi149=x' 2024-03-24T11:30:39.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:38 smithi149 ceph-mon[25685]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:30:39.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:38 smithi149 ceph-mon[25685]: Reconfiguring daemon mon.c on smithi145 2024-03-24T11:30:39.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:38 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2027167519' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:39.242 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:38 smithi145 ceph-mon[24853]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:30:39.242 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:38 smithi145 ceph-mon[24853]: Reconfiguring daemon mon.c on smithi145 2024-03-24T11:30:39.242 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:38 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2027167519' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:39.243 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:38 smithi145 ceph-mon[29394]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:30:39.243 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:38 smithi145 ceph-mon[29394]: Reconfiguring daemon mon.c on smithi145 2024-03-24T11:30:39.243 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:38 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2027167519' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:40.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:39 smithi149 ceph-mon[25685]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:40.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:39 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:40.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:39 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:40.186 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:39 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:40.186 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:39 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[24853]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:40.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[29394]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:30:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:30:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:39 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:41.009 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:40 smithi149 ceph-mon[25685]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:30:41.009 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:40 smithi149 ceph-mon[25685]: Reconfiguring daemon mon.b on smithi149 2024-03-24T11:30:41.305 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled mgr update... 2024-03-24T11:30:41.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:40 smithi145 ceph-mon[24853]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:30:41.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:40 smithi145 ceph-mon[24853]: Reconfiguring daemon mon.b on smithi149 2024-03-24T11:30:41.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:40 smithi145 ceph-mon[29394]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:30:41.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:40 smithi145 ceph-mon[29394]: Reconfiguring daemon mon.b on smithi149 2024-03-24T11:30:42.064 DEBUG:teuthology.orchestra.run.smithi149:mgr.x> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T11:30:42.067 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:30:42.067 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:30:42.067 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:30:42.086 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:30:42.087 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-03-24T11:30:42.143 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:30:42.143 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:42.143 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-24T11:30:42.144 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:42.144 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:42.144 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-24 11:30:18.583104658 +0000 2024-03-24T11:30:42.144 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-24 11:28:04.989322743 +0000 2024-03-24T11:30:42.144 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-24 11:28:04.989322743 +0000 2024-03-24T11:30:42.144 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-24 11:28:04.989322743 +0000 2024-03-24T11:30:42.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:30:42.211 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:30:42.212 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:42.214 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-24T11:30:42.214 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-24T11:30:42.214 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000220567 s, 2.3 MB/s 2024-03-24T11:30:42.215 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:30:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:30:42.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:30:42.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:42.273 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-24 11:30:18.583104658 +0000 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-24 11:28:05.326315558 +0000 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-24 11:28:05.326315558 +0000 2024-03-24T11:30:42.330 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-24 11:28:05.326315558 +0000 2024-03-24T11:30:42.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:30:42.399 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-24T11:30:42.399 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-24T11:30:42.399 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000210513 s, 2.4 MB/s 2024-03-24T11:30:42.400 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:30:42.458 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-03-24T11:30:42.515 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-24 11:30:18.584104637 +0000 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-24 11:28:05.661308415 +0000 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-24 11:28:05.661308415 +0000 2024-03-24T11:30:42.516 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-24 11:28:05.661308415 +0000 2024-03-24T11:30:42.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:30:42.584 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-24T11:30:42.585 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-24T11:30:42.585 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000206454 s, 2.5 MB/s 2024-03-24T11:30:42.586 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:30:42.643 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-24 11:30:18.584104637 +0000 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-24 11:28:05.995301293 +0000 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-24 11:28:05.995301293 +0000 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-24 11:28:05.995301293 +0000 2024-03-24T11:30:42.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000230111 s, 2.2 MB/s 2024-03-24T11:30:42.770 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:30:42.827 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-24T11:30:42.827 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:30:42.844 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:30:42.845 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2024-03-24T11:30:42.901 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:30:42.901 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:42.901 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T11:30:42.901 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:42.901 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:42.901 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-24 11:30:27.861224752 +0000 2024-03-24T11:30:42.901 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-24 11:28:05.086241836 +0000 2024-03-24T11:30:42.902 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-24 11:28:05.086241836 +0000 2024-03-24T11:30:42.902 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-24 11:28:05.086241836 +0000 2024-03-24T11:30:42.902 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:30:42.971 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-24T11:30:42.971 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-24T11:30:42.971 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000273526 s, 1.9 MB/s 2024-03-24T11:30:42.972 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:30:43.029 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-24 11:30:27.861224752 +0000 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-24 11:28:05.415235283 +0000 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-24 11:28:05.415235283 +0000 2024-03-24T11:30:43.085 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-24 11:28:05.415235283 +0000 2024-03-24T11:30:43.086 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:30:43.152 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-24T11:30:43.152 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-24T11:30:43.152 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000217713 s, 2.4 MB/s 2024-03-24T11:30:43.154 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:30:43.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:42 smithi149 ceph-mon[25685]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi145=y;smithi149=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:43.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:42 smithi149 ceph-mon[25685]: Saving service mgr spec with placement smithi145=y;smithi149=x;count:2 2024-03-24T11:30:43.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:42 smithi149 ceph-mon[25685]: Deploying daemon mgr.x on smithi149 2024-03-24T11:30:43.183 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-24 11:30:27.861224752 +0000 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-24 11:28:05.757228472 +0000 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-24 11:28:05.757228472 +0000 2024-03-24T11:30:43.240 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-24 11:28:05.757228472 +0000 2024-03-24T11:30:43.241 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:30:43.317 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-24T11:30:43.317 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-24T11:30:43.317 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000213428 s, 2.4 MB/s 2024-03-24T11:30:43.318 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:30:43.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:42 smithi145 ceph-mon[24853]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi145=y;smithi149=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:43.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:42 smithi145 ceph-mon[24853]: Saving service mgr spec with placement smithi145=y;smithi149=x;count:2 2024-03-24T11:30:43.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:42 smithi145 ceph-mon[24853]: Deploying daemon mgr.x on smithi149 2024-03-24T11:30:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:42 smithi145 ceph-mon[29394]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi145=y;smithi149=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:42 smithi145 ceph-mon[29394]: Saving service mgr spec with placement smithi145=y;smithi149=x;count:2 2024-03-24T11:30:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:42 smithi145 ceph-mon[29394]: Deploying daemon mgr.x on smithi149 2024-03-24T11:30:43.373 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2024-03-24T11:30:43.428 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:30:43.428 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:30:43.428 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T11:30:43.429 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:30:43.429 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:30:43.429 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-24 11:30:43.274921434 +0000 2024-03-24T11:30:43.429 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-24 11:28:06.099221660 +0000 2024-03-24T11:30:43.429 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-24 11:28:06.099221660 +0000 2024-03-24T11:30:43.429 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-24 11:28:06.099221660 +0000 2024-03-24T11:30:43.429 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:30:43.499 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-24T11:30:43.499 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-24T11:30:43.499 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000241405 s, 2.1 MB/s 2024-03-24T11:30:43.500 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:30:43.558 INFO:tasks.cephadm:Deploying osd.0 on smithi145 with /dev/vg_nvme/lv_4... 2024-03-24T11:30:43.558 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:30:44.082 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:43 smithi149 ceph-mon[25685]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:44.082 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:44 smithi149 systemd[1]: Started Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:30:44.156 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:43 smithi145 ceph-mon[24853]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:44.156 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:43 smithi145 ceph-mon[29394]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:44.767 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:44.476+0000 7f91486ea000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:30:44.767 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:44.565+0000 7f91486ea000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:30:45.354 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:45.354 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:45.354 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:45.354 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:45.354 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-mon[25685]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:45.354 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.183+0000 7f91486ea000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:30:45.354 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.270+0000 7f91486ea000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:30:45.355 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.353+0000 7f91486ea000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:30:45.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[24853]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:45 smithi145 ceph-mon[29394]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:45.569 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:30:45.569 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:30:45.569 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-24T11:30:45.569 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-24T11:30:45.569 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115281 s, 910 MB/s 2024-03-24T11:30:45.569 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-24T11:30:45.692 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.503+0000 7f91486ea000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:30:45.692 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.583+0000 7f91486ea000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:30:45.692 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.691+0000 7f91486ea000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:30:45.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.774+0000 7f91486ea000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:30:45.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:45.939+0000 7f91486ea000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:30:46.178 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: Reconfiguring daemon mgr.y on smithi145 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:30:46.179 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:30:46.180 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:46.180 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: Reconfiguring daemon mgr.y on smithi145 2024-03-24T11:30:46.180 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:46 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:46.024+0000 7f91486ea000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:30:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:30:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:30:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:30:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:46.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: Reconfiguring daemon mgr.y on smithi145 2024-03-24T11:30:46.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:46 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:46.845 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_4 2024-03-24T11:30:47.381 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:47.062+0000 7f91486ea000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:30:47.381 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:47.190+0000 7f91486ea000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:30:47.381 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:47.380+0000 7f91486ea000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:30:47.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:47.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:47.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:47.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:47.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:47.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-mon[25685]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:47.726 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:47.467+0000 7f91486ea000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:30:47.726 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:47.602+0000 7f91486ea000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:30:47.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:47.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:47.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[24853]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:30:47.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:47 smithi145 ceph-mon[29394]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:48.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:47.925+0000 7f91486ea000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:30:48.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:48.142+0000 7f91486ea000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:30:48.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:48.225+0000 7f91486ea000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:30:48.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:48.392+0000 7f91486ea000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:30:49.058 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:48 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:30:49.058 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:48 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:30:49.058 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:48 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:49.058 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:48 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:30:49.058 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:48 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:30:49.058 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:48 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:48 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:30:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:48 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:30:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:48 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:49.606 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:49.315+0000 7f91486ea000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:30:49.607 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:49.605+0000 7f91486ea000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:30:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:49 smithi149 ceph-mon[25685]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:49 smithi149 ceph-mon[25685]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:49.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:49.700+0000 7f91486ea000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:30:49.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:30:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:30:49.851+0000 7f91486ea000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:30:49.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:49 smithi145 ceph-mon[24853]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:49.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:49 smithi145 ceph-mon[24853]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:49.988 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:49 smithi145 ceph-mon[29394]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:30:49.988 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:49 smithi145 ceph-mon[29394]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:51.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[24853]: Standby manager daemon x started 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[29394]: Standby manager daemon x started 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:30:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:50 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:30:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:50 smithi149 ceph-mon[25685]: Standby manager daemon x started 2024-03-24T11:30:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:50 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:30:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:50 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:30:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:50 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:30:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:50 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1348708066' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:30:52.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/699856078' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]: dispatch 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]: dispatch 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]': finished 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3338735803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:30:52.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/699856078' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]: dispatch 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]: dispatch 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]': finished 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:30:52.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:51 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3338735803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/699856078' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]: dispatch 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]: dispatch 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f61b5c1-0892-46a5-b37d-403044def119"}]': finished 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:30:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:30:52.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:51 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3338735803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:30:54.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:53 smithi145 ceph-mon[24853]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:54.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:53 smithi145 ceph-mon[29394]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:54.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:53 smithi149 ceph-mon[25685]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:56.134 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:55 smithi145 ceph-mon[29394]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:56.134 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:55 smithi145 ceph-mon[24853]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:55 smithi149 ceph-mon[25685]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:58.149 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:57 smithi145 ceph-mon[24853]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:58.149 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:30:58.149 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:58.149 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:57 smithi145 ceph-mon[29394]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:58.150 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:30:58.150 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:57 smithi149 ceph-mon[25685]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:30:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:30:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:59.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:58 smithi145 ceph-mon[24853]: Deploying daemon osd.0 on smithi145 2024-03-24T11:30:59.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:58 smithi145 ceph-mon[29394]: Deploying daemon osd.0 on smithi145 2024-03-24T11:30:59.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:58 smithi149 ceph-mon[25685]: Deploying daemon osd.0 on smithi145 2024-03-24T11:31:00.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:30:59 smithi145 ceph-mon[24853]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:00.198 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:30:59 smithi145 ceph-mon[29394]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:00.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:30:59 smithi149 ceph-mon[25685]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:02.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[29394]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:02.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:02.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:02.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:02.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:02.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[24853]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:02.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:02.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:02.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:02.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:01 smithi149 ceph-mon[25685]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:04.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:03 smithi145 ceph-mon[24853]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:03 smithi145 ceph-mon[29394]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:03 smithi149 ceph-mon[25685]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:04.731 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 0 on host 'smithi145' 2024-03-24T11:31:05.276 DEBUG:teuthology.orchestra.run.smithi145:osd.0> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:31:05.278 INFO:tasks.cephadm:Deploying osd.1 on smithi145 with /dev/vg_nvme/lv_3... 2024-03-24T11:31:05.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:31:05.899 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:05.899 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[24853]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:05.899 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:05.899 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[24853]: from='osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:31:05.899 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[24853]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:31:05.900 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:05.900 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[29394]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:05.900 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:05.900 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[29394]: from='osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:31:05.900 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:05 smithi145 ceph-mon[29394]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:31:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:05 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:05 smithi149 ceph-mon[25685]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:05 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:05 smithi149 ceph-mon[25685]: from='osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:31:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:05 smithi149 ceph-mon[25685]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:31:07.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[24853]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[24853]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[24853]: from='osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[24853]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[24853]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[29394]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[29394]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:07.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[29394]: from='osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:07.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[29394]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:07.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-mon[29394]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:07.113 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[32260]: 2024-03-24T11:31:06.985+0000 7ffbbdf2a700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T11:31:07.113 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[32260]: 2024-03-24T11:31:06.987+0000 7ffbb68bd700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:31:07.282 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:31:07.282 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:31:07.282 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-24T11:31:07.283 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-24T11:31:07.283 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0188386 s, 557 MB/s 2024-03-24T11:31:07.283 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-24T11:31:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:06 smithi149 ceph-mon[25685]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:31:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:06 smithi149 ceph-mon[25685]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:31:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:06 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:06 smithi149 ceph-mon[25685]: from='osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:06 smithi149 ceph-mon[25685]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:06 smithi149 ceph-mon[25685]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:08.240 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[29394]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[29394]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:31:08.241 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[24853]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:08.242 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[24853]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:31:08.242 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:08.242 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:07 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:07 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:31:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:07 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:31:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:07 smithi149 ceph-mon[25685]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:07 smithi149 ceph-mon[25685]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:31:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:07 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:07 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:08.625 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_3 2024-03-24T11:31:09.195 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:09 smithi145 ceph-mon[29394]: osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160] boot 2024-03-24T11:31:09.195 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:09 smithi145 ceph-mon[29394]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:31:09.195 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:09 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:09.195 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:09 smithi145 ceph-mon[29394]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:09.196 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:08 smithi145 ceph-mon[24853]: osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160] boot 2024-03-24T11:31:09.196 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:08 smithi145 ceph-mon[24853]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:31:09.196 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:09.196 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:08 smithi145 ceph-mon[24853]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:09.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:08 smithi149 ceph-mon[25685]: osd.0 [v2:172.21.15.145:6802/488109160,v1:172.21.15.145:6803/488109160] boot 2024-03-24T11:31:09.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:08 smithi149 ceph-mon[25685]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:31:09.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:08 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:31:09.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:08 smithi149 ceph-mon[25685]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:31:11.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:31:11.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi145 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi145 to 16064M 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[24853]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:11.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi145 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi145 to 16064M 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:11 smithi145 ceph-mon[29394]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:31:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi145 2024-03-24T11:31:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi145 to 16064M 2024-03-24T11:31:11.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:11.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:11.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:11.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:11.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:11 smithi149 ceph-mon[25685]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:12.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[29394]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:12.264 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:12.264 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:12.264 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:12.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[24853]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:12.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:12.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:12.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:12 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:12 smithi149 ceph-mon[25685]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:12 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:12 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:12 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[24853]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2757251629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]: dispatch 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]: dispatch 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]': finished 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[24853]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[29394]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2757251629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]: dispatch 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]: dispatch 2024-03-24T11:31:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]': finished 2024-03-24T11:31:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[29394]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:31:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:13 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:13 smithi149 ceph-mon[25685]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:13 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2757251629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]: dispatch 2024-03-24T11:31:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:13 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]: dispatch 2024-03-24T11:31:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:13 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "564e785b-94b7-4f54-a7d1-d65101b260d5"}]': finished 2024-03-24T11:31:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:13 smithi149 ceph-mon[25685]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:31:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:13 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:15.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:14 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/128399973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:31:15.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:14 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/128399973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:31:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:14 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/128399973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:31:16.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:15 smithi145 ceph-mon[29394]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:16.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:15 smithi145 ceph-mon[24853]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:15 smithi149 ceph-mon[25685]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:18.214 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:17 smithi145 ceph-mon[24853]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:18.214 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:17 smithi145 ceph-mon[29394]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:17 smithi149 ceph-mon[25685]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:19 smithi149 ceph-mon[25685]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:19 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:31:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:19 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:20.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:19 smithi145 ceph-mon[24853]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:20.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:19 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:31:20.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:19 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:20.264 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:19 smithi145 ceph-mon[29394]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:20.264 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:19 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:31:20.264 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:19 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:21.085 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:20 smithi145 ceph-mon[24853]: Deploying daemon osd.1 on smithi145 2024-03-24T11:31:21.086 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:20 smithi145 ceph-mon[29394]: Deploying daemon osd.1 on smithi145 2024-03-24T11:31:21.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:20 smithi149 ceph-mon[25685]: Deploying daemon osd.1 on smithi145 2024-03-24T11:31:22.218 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:21 smithi145 ceph-mon[24853]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:22.218 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:21 smithi145 ceph-mon[29394]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:22.224 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:21 smithi149 ceph-mon[25685]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:23.898 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:23 smithi145 ceph-mon[29394]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:23.899 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:23 smithi145 ceph-mon[24853]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:23 smithi149 ceph-mon[25685]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[29394]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:25.692 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:25.693 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:25 smithi145 ceph-mon[24853]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:25.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:25.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:25.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:25.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:25.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:25 smithi149 ceph-mon[25685]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:27.531 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 1 on host 'smithi145' 2024-03-24T11:31:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[24853]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[24853]: from='osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:31:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[24853]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:31:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[29394]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[29394]: from='osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:31:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:27 smithi145 ceph-mon[29394]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:31:28.164 DEBUG:teuthology.orchestra.run.smithi145:osd.1> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:31:28.166 INFO:tasks.cephadm:Deploying osd.2 on smithi145 with /dev/vg_nvme/lv_2... 2024-03-24T11:31:28.166 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:31:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:27 smithi149 ceph-mon[25685]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:27 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:27 smithi149 ceph-mon[25685]: from='osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:31:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:27 smithi149 ceph-mon[25685]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:31:29.445 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:29.445 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[29394]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:31:29.445 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[29394]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:31:29.445 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:29.445 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[29394]: from='osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:29.445 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[29394]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:29.445 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[29394]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:29.446 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:29.446 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[24853]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:31:29.446 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[24853]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:31:29.446 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:29.446 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[24853]: from='osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:29.446 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[24853]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:29.446 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-mon[24853]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:29 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:29 smithi149 ceph-mon[25685]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:31:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:29 smithi149 ceph-mon[25685]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:31:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:29 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:29 smithi149 ceph-mon[25685]: from='osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:29 smithi149 ceph-mon[25685]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:29 smithi149 ceph-mon[25685]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:29.774 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[35228]: 2024-03-24T11:31:29.539+0000 7f3798a7f700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T11:31:29.774 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:29 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[35228]: 2024-03-24T11:31:29.543+0000 7f3792414700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:31:30.719 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[24853]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:30.719 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[24853]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:31:30.719 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:30.719 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:30.719 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[29394]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:30.720 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[29394]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:31:30.720 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:30.720 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:30.864 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:30 smithi149 ceph-mon[25685]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:30.864 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:30 smithi149 ceph-mon[25685]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:31:30.865 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:30.865 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:31.571 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:31:31.571 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:31:31.571 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-24T11:31:31.571 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-24T11:31:31.572 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0163701 s, 641 MB/s 2024-03-24T11:31:31.572 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-24T11:31:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:31:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[24853]: osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099] boot 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[24853]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[24853]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[29394]: osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099] boot 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[29394]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:31:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:31.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:31 smithi145 ceph-mon[29394]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:31 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:31:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:31 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:31:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:31 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:31 smithi149 ceph-mon[25685]: osd.1 [v2:172.21.15.145:6810/544022099,v1:172.21.15.145:6811/544022099] boot 2024-03-24T11:31:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:31 smithi149 ceph-mon[25685]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:31:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:31 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:31:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:31 smithi149 ceph-mon[25685]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:31:32.280 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_2 2024-03-24T11:31:32.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:32 smithi145 ceph-mon[24853]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:31:32.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:32 smithi145 ceph-mon[29394]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:31:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:32 smithi149 ceph-mon[25685]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi145 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi145 to 8032M 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:33.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[24853]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi145 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi145 to 8032M 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.825 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.826 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:33.826 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:33.826 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:33.826 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.826 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:33 smithi145 ceph-mon[29394]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi145 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi145 to 8032M 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:33.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:33 smithi149 ceph-mon[25685]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:34.988 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:34 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:34.988 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:34 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:34.988 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:34 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:34.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:34 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:34.989 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:34 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:34.989 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:34 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:34 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:34 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:34 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:35.947 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:35 smithi145 ceph-mon[29394]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:35.947 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:35 smithi145 ceph-mon[29394]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:35.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:35 smithi145 ceph-mon[24853]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:35.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:35 smithi145 ceph-mon[24853]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:35 smithi149 ceph-mon[25685]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:35 smithi149 ceph-mon[25685]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:37.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4045959221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4f6183d-f2de-46bf-ac81-271da6c29463"}]: dispatch 2024-03-24T11:31:37.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4045959221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4f6183d-f2de-46bf-ac81-271da6c29463"}]': finished 2024-03-24T11:31:37.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[24853]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:31:37.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:37.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4045959221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4f6183d-f2de-46bf-ac81-271da6c29463"}]: dispatch 2024-03-24T11:31:37.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4045959221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4f6183d-f2de-46bf-ac81-271da6c29463"}]': finished 2024-03-24T11:31:37.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[29394]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:31:37.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:36 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:36 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4045959221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4f6183d-f2de-46bf-ac81-271da6c29463"}]: dispatch 2024-03-24T11:31:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:36 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4045959221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4f6183d-f2de-46bf-ac81-271da6c29463"}]': finished 2024-03-24T11:31:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:36 smithi149 ceph-mon[25685]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:31:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:36 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:38.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:37 smithi145 ceph-mon[24853]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:38.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:37 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2949694830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:31:38.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:37 smithi145 ceph-mon[29394]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:38.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:37 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2949694830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:31:38.224 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:37 smithi149 ceph-mon[25685]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:37 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2949694830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:31:40.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:39 smithi145 ceph-mon[24853]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:40.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:39 smithi145 ceph-mon[29394]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:39 smithi149 ceph-mon[25685]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:42.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:41 smithi145 ceph-mon[24853]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:42.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:41 smithi145 ceph-mon[29394]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:41 smithi149 ceph-mon[25685]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:43.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[29394]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:43.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:31:43.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:43.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[29394]: Deploying daemon osd.2 on smithi145 2024-03-24T11:31:43.911 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[24853]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:43.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:31:43.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:43.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:43 smithi145 ceph-mon[24853]: Deploying daemon osd.2 on smithi145 2024-03-24T11:31:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:43 smithi149 ceph-mon[25685]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:43 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:31:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:43 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:43 smithi149 ceph-mon[25685]: Deploying daemon osd.2 on smithi145 2024-03-24T11:31:46.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:45 smithi145 ceph-mon[24853]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:46.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:45 smithi145 ceph-mon[29394]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:45 smithi149 ceph-mon[25685]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:48.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[24853]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:48.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:48.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:48.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:48.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:48.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[29394]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:48.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:48.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:48.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:48.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:47 smithi149 ceph-mon[25685]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:50.083 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:49 smithi145 ceph-mon[24853]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:50.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:49 smithi145 ceph-mon[29394]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:50.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:49 smithi149 ceph-mon[25685]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:51.081 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 2 on host 'smithi145' 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:51.665 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:51.666 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:51.688 DEBUG:teuthology.orchestra.run.smithi145:osd.2> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:31:51.690 INFO:tasks.cephadm:Deploying osd.3 on smithi145 with /dev/vg_nvme/lv_1... 2024-03-24T11:31:51.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:52.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:52 smithi149 ceph-mon[25685]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:31:52.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:52 smithi149 ceph-mon[25685]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:31:52.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:52 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:52.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:52 smithi149 ceph-mon[25685]: from='osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:52.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:52 smithi149 ceph-mon[25685]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[29394]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[29394]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[29394]: from='osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[29394]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[24853]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[24853]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:31:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[24853]: from='osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-mon[24853]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:31:52.863 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[38148]: 2024-03-24T11:31:52.376+0000 7fd15bf35700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T11:31:52.863 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:52 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[38148]: 2024-03-24T11:31:52.379+0000 7fd1560cb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:31:53.591 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[29394]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[29394]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[29394]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[24853]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[24853]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:53.592 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:53 smithi145 ceph-mon[24853]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:53 smithi149 ceph-mon[25685]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:31:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:53 smithi149 ceph-mon[25685]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:31:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:53 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:53 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:53 smithi149 ceph-mon[25685]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:31:54.426 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:31:54.427 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:31:54.427 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-24T11:31:54.427 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-24T11:31:54.427 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124552 s, 842 MB/s 2024-03-24T11:31:54.427 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-24T11:31:54.692 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[24853]: osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660] boot 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[24853]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[29394]: osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660] boot 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[29394]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:31:54.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:54 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:54 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:31:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:54 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:31:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:54 smithi149 ceph-mon[25685]: osd.2 [v2:172.21.15.145:6818/1601008660,v1:172.21.15.145:6819/1601008660] boot 2024-03-24T11:31:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:54 smithi149 ceph-mon[25685]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:31:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:54 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:31:55.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_1 2024-03-24T11:31:56.086 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:55 smithi145 ceph-mon[24853]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:31:56.087 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:55 smithi145 ceph-mon[24853]: pgmap v71: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:31:56.087 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T11:31:56.087 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:55 smithi145 ceph-mon[29394]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:31:56.087 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:55 smithi145 ceph-mon[29394]: pgmap v71: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:31:56.087 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T11:31:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:55 smithi149 ceph-mon[25685]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:31:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:55 smithi149 ceph-mon[25685]: pgmap v71: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:31:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi145 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi145 to 5354M 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:56.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi145 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi145 to 5354M 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:56.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:56 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T11:31:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:31:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:31:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi145 2024-03-24T11:31:57.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:57.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:57.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:57.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:31:57.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi145 to 5354M 2024-03-24T11:31:57.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:57.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:56 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:57.536 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39468]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:31:57.536 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39468]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:57.536 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39468]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:57.536 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39493]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:31:57.813 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39493]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:57.813 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39493]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:58.084 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39533]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:31:58.084 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39533]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:58.084 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:57 smithi145 sudo[39533]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:58.085 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:31:58.085 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[29394]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:31:58.085 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[29394]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:31:58.085 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:58.085 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:58.085 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:58.085 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:58.086 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:31:58.086 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[24853]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:31:58.086 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[24853]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:31:58.086 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:58.086 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:58.087 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:58.087 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:31:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:57 smithi149 ceph-mon[25685]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:31:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:57 smithi149 ceph-mon[25685]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:31:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:31:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:31:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:31:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:58.338 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39579]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:31:58.338 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39579]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:58.338 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39579]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:58.594 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39600]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:31:58.594 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39600]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:58.594 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39600]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:58.852 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39605]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:31:58.852 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39605]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:58.852 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39605]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:58.852 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39609]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:31:58.852 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39609]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:58.852 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:58 smithi145 sudo[39609]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:59.104 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39631]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:31:59.104 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39631]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:59.105 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39631]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:31:59.356 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[29394]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:59.356 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[29394]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:31:59.357 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[29394]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:31:59.357 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[29394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:31:59.357 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[24853]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:59.357 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[24853]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:31:59.357 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[24853]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:31:59.357 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:31:59 smithi145 ceph-mon[24853]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:31:59.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:59 smithi149 ceph-mon[25685]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:31:59.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:59 smithi149 ceph-mon[25685]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:31:59.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:59 smithi149 ceph-mon[25685]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:31:59.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:59 smithi149 ceph-mon[25685]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:31:59.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39640]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:31:59.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39640]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:31:59.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39640]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:32:00.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39656]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:32:00.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39656]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:32:00.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:31:59 smithi145 sudo[39656]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:32:00.113 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:59 smithi149 sudo[27211]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:32:00.113 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:59 smithi149 sudo[27211]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:32:00.113 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:31:59 smithi149 sudo[27211]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:32:00.114 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 sudo[27214]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 sudo[27214]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 sudo[27214]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:32:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:00 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/443086489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "580cea31-9ffe-44e7-bdab-d36551219e6c"}]: dispatch 2024-03-24T11:32:00.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/443086489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "580cea31-9ffe-44e7-bdab-d36551219e6c"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:32:00.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:00 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/443086489' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "580cea31-9ffe-44e7-bdab-d36551219e6c"}]: dispatch 2024-03-24T11:32:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:32:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/443086489' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "580cea31-9ffe-44e7-bdab-d36551219e6c"}]': finished 2024-03-24T11:32:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:32:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/438546624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:01.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:01 smithi149 ceph-mon[25685]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:01.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:32:01.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:01.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/443086489' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "580cea31-9ffe-44e7-bdab-d36551219e6c"}]': finished 2024-03-24T11:32:01.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/438546624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[24853]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/443086489' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "580cea31-9ffe-44e7-bdab-d36551219e6c"}]': finished 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/438546624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:01.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:01 smithi145 ceph-mon[29394]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:04.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:03 smithi145 ceph-mon[29394]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:04.099 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:03 smithi145 ceph-mon[24853]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:03 smithi149 ceph-mon[25685]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:06.110 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:05 smithi145 ceph-mon[29394]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:06.110 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:05 smithi145 ceph-mon[24853]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:05 smithi149 ceph-mon[25685]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:07.061 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:06 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:32:07.061 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:06 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:07.062 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:06 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:32:07.062 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:06 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:07.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:06 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:32:07.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:06 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:07.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:07 smithi145 ceph-mon[24853]: Deploying daemon osd.3 on smithi145 2024-03-24T11:32:07.997 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:07 smithi145 ceph-mon[24853]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:07.997 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:07 smithi145 ceph-mon[29394]: Deploying daemon osd.3 on smithi145 2024-03-24T11:32:07.997 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:07 smithi145 ceph-mon[29394]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:07 smithi149 ceph-mon[25685]: Deploying daemon osd.3 on smithi145 2024-03-24T11:32:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:07 smithi149 ceph-mon[25685]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:10.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:09 smithi145 ceph-mon[29394]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:09 smithi145 ceph-mon[24853]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:09 smithi149 ceph-mon[25685]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[29394]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[24853]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:11.970 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:11.971 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:11 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:11 smithi149 ceph-mon[25685]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:11 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:14.078 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:13 smithi145 ceph-mon[29394]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:14.078 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:13 smithi145 ceph-mon[24853]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:13 smithi149 ceph-mon[25685]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:14.708 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 3 on host 'smithi145' 2024-03-24T11:32:15.348 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.348 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:32:15.348 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[24853]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:15.349 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:15.350 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:15.350 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:15 smithi145 ceph-mon[29394]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:15.370 DEBUG:teuthology.orchestra.run.smithi145:osd.3> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T11:32:15.372 INFO:tasks.cephadm:Deploying osd.4 on smithi149 with /dev/vg_nvme/lv_4... 2024-03-24T11:32:15.372 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:32:15.402 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.402 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:32:15.403 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.403 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:15.403 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:15.403 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:15.403 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:15.403 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:15 smithi149 ceph-mon[25685]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:16 smithi149 ceph-mon[25685]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:32:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:16 smithi149 ceph-mon[25685]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:32:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:16 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:16 smithi149 ceph-mon[25685]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:32:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:16 smithi149 ceph-mon[25685]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:32:16.487 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[24853]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:32:16.487 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[24853]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[24853]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[24853]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[29394]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[29394]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[29394]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:32:16.488 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-mon[29394]: from='osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-24T11:32:16.488 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[41123]: 2024-03-24T11:32:16.154+0000 7fcaefaa5700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T11:32:16.489 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:32:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[41123]: 2024-03-24T11:32:16.159+0000 7fcae9c3b700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:32:17.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:17 smithi149 ceph-mon[25685]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:32:17.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:17 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:17.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:17 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:17.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:17 smithi149 ceph-mon[25685]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:17.461 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:32:17.461 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:32:17.461 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-03-24T11:32:17.461 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-03-24T11:32:17.461 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011516 s, 911 MB/s 2024-03-24T11:32:17.461 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-03-24T11:32:17.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[29394]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:32:17.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:17.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:17.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[29394]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:17.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[24853]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:32:17.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:17.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:17.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:17 smithi145 ceph-mon[24853]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:32:18.048 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_4 2024-03-24T11:32:18.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490] boot 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490] boot 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:18 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.406 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:32:18.406 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:32:18.406 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:18.406 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: osd.3 [v2:172.21.15.145:6826/3558706490,v1:172.21.15.145:6827/3558706490] boot 2024-03-24T11:32:18.406 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:32:18.406 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:32:18.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:18.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:18.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:18 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:19.372 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:19 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi145 2024-03-24T11:32:19.372 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:19 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi145 to 4016M 2024-03-24T11:32:19.373 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:19 smithi149 ceph-mon[25685]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:32:19.373 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:19 smithi149 ceph-mon[25685]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:19.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi145 2024-03-24T11:32:19.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi145 to 4016M 2024-03-24T11:32:19.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[29394]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:32:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[29394]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:19.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi145 2024-03-24T11:32:19.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi145 to 4016M 2024-03-24T11:32:19.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[24853]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:32:19.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:19 smithi145 ceph-mon[24853]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:20.394 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:20 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:32:20.395 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:20 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:32:20.395 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:20 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:20.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:20 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:32:20.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:20 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:32:20.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:20 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:20.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:20 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:32:20.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:20 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:32:20.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:20 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:21.307 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:21 smithi149 ceph-mon[25685]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:32:21.307 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:21 smithi149 ceph-mon[25685]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:21.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:21 smithi145 ceph-mon[24853]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:32:21.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:21 smithi145 ceph-mon[24853]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:21.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:21 smithi145 ceph-mon[29394]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:32:21.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:21 smithi145 ceph-mon[29394]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:22.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/2278160072' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]: dispatch 2024-03-24T11:32:22.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]: dispatch 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]': finished 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[29394]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/2278160072' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]: dispatch 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]: dispatch 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]': finished 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[24853]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:32:22.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:22 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:22 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/2278160072' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]: dispatch 2024-03-24T11:32:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:22 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]: dispatch 2024-03-24T11:32:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:22 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a6d81d6-ffae-47b5-af8d-b22437386ff8"}]': finished 2024-03-24T11:32:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:22 smithi149 ceph-mon[25685]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:32:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:22 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:24.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:23 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/1158277607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:24.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:23 smithi145 ceph-mon[29394]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:23 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/1158277607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:23 smithi145 ceph-mon[24853]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:23 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/1158277607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:23 smithi149 ceph-mon[25685]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:25 smithi149 ceph-mon[25685]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:26.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:25 smithi145 ceph-mon[24853]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:26.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:25 smithi145 ceph-mon[29394]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:28.011 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:27 smithi149 ceph-mon[25685]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:28.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:27 smithi145 ceph-mon[24853]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:28.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:27 smithi145 ceph-mon[29394]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:29.090 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:28 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:32:29.091 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:28 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:29.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:28 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:32:29.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:28 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:29.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:28 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:32:29.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:28 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:30.188 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:29 smithi149 ceph-mon[25685]: Deploying daemon osd.4 on smithi149 2024-03-24T11:32:30.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:29 smithi149 ceph-mon[25685]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:29 smithi145 ceph-mon[24853]: Deploying daemon osd.4 on smithi149 2024-03-24T11:32:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:29 smithi145 ceph-mon[24853]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:29 smithi145 ceph-mon[29394]: Deploying daemon osd.4 on smithi149 2024-03-24T11:32:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:29 smithi145 ceph-mon[29394]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:32.105 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:31 smithi149 ceph-mon[25685]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:32.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:31 smithi145 ceph-mon[24853]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:32.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:31 smithi145 ceph-mon[29394]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:33.965 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:33 smithi149 ceph-mon[25685]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:33.965 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:33.965 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:33.965 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:33.965 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[24853]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[29394]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:35 smithi149 ceph-mon[25685]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:36.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:35 smithi145 ceph-mon[29394]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:36.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:36.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:36.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:35 smithi145 ceph-mon[24853]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:36.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:36.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:36.641 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 4 on host 'smithi149' 2024-03-24T11:32:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:36 smithi149 ceph-mon[25685]: from='osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:32:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:36 smithi149 ceph-mon[25685]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:32:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:36 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:36 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:36 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:36 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:37.255 DEBUG:teuthology.orchestra.run.smithi149:osd.4> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T11:32:37.257 INFO:tasks.cephadm:Deploying osd.5 on smithi149 with /dev/vg_nvme/lv_3... 2024-03-24T11:32:37.257 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:32:37.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[24853]: from='osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:32:37.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[24853]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:32:37.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[29394]: from='osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[29394]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:37.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:36 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:38.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-mon[25685]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:38.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-mon[25685]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:32:38.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-mon[25685]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:32:38.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:38.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-mon[25685]: from='osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:32:38.213 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-mon[25685]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:32:38.213 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[29136]: 2024-03-24T11:32:37.946+0000 7f9a45580700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T11:32:38.213 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:32:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[29136]: 2024-03-24T11:32:37.950+0000 7f9a3e714700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:32:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[29394]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[29394]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:32:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[29394]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:32:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[29394]: from='osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[29394]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[24853]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[24853]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[24853]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[24853]: from='osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:32:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:37 smithi145 ceph-mon[24853]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:32:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:38 smithi149 ceph-mon[25685]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:32:38.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:38 smithi149 ceph-mon[25685]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:32:38.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:38 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:38.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:38 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:38.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:38 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:39.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[29394]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[29394]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[24853]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[24853]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:39.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:38 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:40.115 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:32:40.115 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:32:40.115 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-03-24T11:32:40.115 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-03-24T11:32:40.115 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137418 s, 763 MB/s 2024-03-24T11:32:40.115 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-03-24T11:32:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:39 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:32:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:39 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:32:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:39 smithi149 ceph-mon[25685]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:39 smithi149 ceph-mon[25685]: osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003] boot 2024-03-24T11:32:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:39 smithi149 ceph-mon[25685]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:32:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:39 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:32:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:32:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[24853]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[24853]: osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003] boot 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[24853]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[29394]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[29394]: osd.4 [v2:172.21.15.149:6800/3015006003,v1:172.21.15.149:6801/3015006003] boot 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[29394]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:32:40.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:39 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:32:41.054 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_3 2024-03-24T11:32:41.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:40 smithi149 ceph-mon[25685]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:32:41.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:40 smithi149 ceph-mon[25685]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:32:41.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:40 smithi145 ceph-mon[29394]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:32:41.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:40 smithi145 ceph-mon[29394]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:32:41.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:40 smithi145 ceph-mon[24853]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:32:41.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:40 smithi145 ceph-mon[24853]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:41 smithi149 ceph-mon[25685]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:41 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[29394]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[24853]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:41 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:43.076 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:42 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi149 2024-03-24T11:32:43.077 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:42 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi149 to 5816M 2024-03-24T11:32:43.077 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:42 smithi149 ceph-mon[25685]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:32:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:42 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi149 2024-03-24T11:32:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:42 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi149 to 5816M 2024-03-24T11:32:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:42 smithi145 ceph-mon[29394]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:32:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:42 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi149 2024-03-24T11:32:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:42 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi149 to 5816M 2024-03-24T11:32:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:42 smithi145 ceph-mon[24853]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:32:44.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:43 smithi149 ceph-mon[25685]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-24T11:32:44.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:43 smithi149 ceph-mon[25685]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:32:44.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:43 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:32:44.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:43 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:32:44.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:43 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:44.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[24853]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-24T11:32:44.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[24853]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[29394]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[29394]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:32:44.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:43 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:45.081 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:44 smithi149 ceph-mon[25685]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:32:45.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:44 smithi145 ceph-mon[24853]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:32:45.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:44 smithi145 ceph-mon[29394]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:32:46.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[24853]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T11:32:46.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/1799893422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]: dispatch 2024-03-24T11:32:46.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]: dispatch 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]': finished 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[24853]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[29394]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/1799893422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]: dispatch 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]: dispatch 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]': finished 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[29394]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:32:46.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:45 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:32:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:45 smithi149 ceph-mon[25685]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T11:32:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:45 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/1799893422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]: dispatch 2024-03-24T11:32:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:45 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]: dispatch 2024-03-24T11:32:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:45 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af3286eb-019d-47b0-9faa-e2d816203030"}]': finished 2024-03-24T11:32:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:45 smithi149 ceph-mon[25685]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:32:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:45 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:32:47.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:46 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/1148113532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:47.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:46 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/1148113532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:46 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/1148113532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:32:48.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:47 smithi145 ceph-mon[24853]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-24T11:32:48.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:47 smithi145 ceph-mon[29394]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-24T11:32:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:47 smithi149 ceph-mon[25685]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-24T11:32:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:48 smithi149 ceph-mon[25685]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:49.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:48 smithi145 ceph-mon[24853]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:49.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:48 smithi145 ceph-mon[29394]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:50 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:32:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:50 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:32:51.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:50 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:32:51.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:50 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:32:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:50 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:32:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:50 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:32:52.140 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:51 smithi149 ceph-mon[25685]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:52.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:51 smithi145 ceph-mon[24853]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:52.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:51 smithi145 ceph-mon[29394]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:52 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:32:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:52 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:53.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:52 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:32:53.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:52 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:53.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:52 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:32:53.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:52 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:54.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:53 smithi149 ceph-mon[25685]: Deploying daemon osd.5 on smithi149 2024-03-24T11:32:54.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:53 smithi149 ceph-mon[25685]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:54.262 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:53 smithi145 ceph-mon[29394]: Deploying daemon osd.5 on smithi149 2024-03-24T11:32:54.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:53 smithi145 ceph-mon[29394]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:54.263 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:53 smithi145 ceph-mon[24853]: Deploying daemon osd.5 on smithi149 2024-03-24T11:32:54.263 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:53 smithi145 ceph-mon[24853]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:55 smithi149 ceph-mon[25685]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:56.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:55 smithi145 ceph-mon[24853]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:56.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:55 smithi145 ceph-mon[29394]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:58.137 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:57 smithi149 ceph-mon[25685]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:58.138 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:58.138 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:58.138 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:58.138 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:57 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:58.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[29394]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[24853]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:57 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:00.097 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:32:59 smithi149 ceph-mon[25685]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:00.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:32:59 smithi145 ceph-mon[29394]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:32:59 smithi145 ceph-mon[24853]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:00.880 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 5 on host 'smithi149' 2024-03-24T11:33:01.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:33:01.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:33:01.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:01.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:01.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:01.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:01.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:33:01.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:33:01.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:01.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:01.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:01.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:01.518 DEBUG:teuthology.orchestra.run.smithi149:osd.5> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T11:33:01.520 INFO:tasks.cephadm:Deploying osd.6 on smithi149 with /dev/vg_nvme/lv_2... 2024-03-24T11:33:01.520 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:33:02.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:02 smithi149 ceph-mon[25685]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:33:02.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:02 smithi149 ceph-mon[25685]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:33:02.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:02 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:02.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:02 smithi149 ceph-mon[25685]: from='osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:02.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:02 smithi149 ceph-mon[25685]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:02.476 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:33:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[32010]: 2024-03-24T11:33:02.130+0000 7fc64251b700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T11:33:02.476 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:33:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[32010]: 2024-03-24T11:33:02.135+0000 7fc63eeb6700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:33:02.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[24853]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[24853]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[24853]: from='osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[24853]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[29394]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[29394]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[29394]: from='osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:02.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:02 smithi145 ceph-mon[29394]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332] boot 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.207 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:03 smithi149 ceph-mon[25685]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:03.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:03.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332] boot 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[24853]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: osd.5 [v2:172.21.15.149:6808/4057913332,v1:172.21.15.149:6809/4057913332] boot 2024-03-24T11:33:03.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T11:33:03.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:33:03.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:03 smithi145 ceph-mon[29394]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:33:04.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:04 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:33:04.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:04 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:33:04.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:04 smithi149 ceph-mon[25685]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:33:04.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:04 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:33:04.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:04 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:33:04.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:04 smithi145 ceph-mon[24853]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:33:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:04 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:33:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:04 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:33:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:04 smithi145 ceph-mon[29394]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:33:05.452 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:33:05.453 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:33:05.453 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-03-24T11:33:05.453 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-03-24T11:33:05.453 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112469 s, 932 MB/s 2024-03-24T11:33:05.453 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-03-24T11:33:06.105 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi149 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi149 to 2908M 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.106 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:05 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:33:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi149 2024-03-24T11:33:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi149 to 2908M 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi149 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi149 to 2908M 2024-03-24T11:33:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:05 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:06.128 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_2 2024-03-24T11:33:06.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:06 smithi149 ceph-mon[25685]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:33:06.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:06 smithi149 ceph-mon[25685]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:33:07.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:06 smithi145 ceph-mon[29394]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:33:07.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:06 smithi145 ceph-mon[29394]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:33:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:06 smithi145 ceph-mon[24853]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:33:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:06 smithi145 ceph-mon[24853]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:33:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:07 smithi149 ceph-mon[25685]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:07 smithi149 ceph-mon[25685]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:33:08.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:07 smithi145 ceph-mon[29394]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:08.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:07 smithi145 ceph-mon[29394]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:33:08.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:07 smithi145 ceph-mon[24853]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:08.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:07 smithi145 ceph-mon[24853]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:33:09.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:08 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:33:09.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:08 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:33:09.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:08 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:33:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:33:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:08 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:08 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:33:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:08 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:33:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:08 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:09 smithi149 ceph-mon[25685]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:33:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:09 smithi149 ceph-mon[25685]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:10.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:09 smithi145 ceph-mon[29394]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:33:10.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:09 smithi145 ceph-mon[29394]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:09 smithi145 ceph-mon[24853]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:33:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:09 smithi145 ceph-mon[24853]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:11.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[24853]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[24853]: Cluster is now healthy 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/2139226874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]: dispatch 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]: dispatch 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]': finished 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[24853]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[29394]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[29394]: Cluster is now healthy 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/2139226874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]: dispatch 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]: dispatch 2024-03-24T11:33:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]': finished 2024-03-24T11:33:11.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[29394]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:33:11.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:10 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:11.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:10 smithi149 ceph-mon[25685]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T11:33:11.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:10 smithi149 ceph-mon[25685]: Cluster is now healthy 2024-03-24T11:33:11.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:10 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/2139226874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]: dispatch 2024-03-24T11:33:11.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:10 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]: dispatch 2024-03-24T11:33:11.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:10 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f448d3f-b9a0-4ba1-bea9-ddde2fce734a"}]': finished 2024-03-24T11:33:11.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:10 smithi149 ceph-mon[25685]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T11:33:11.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:10 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:12.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:11 smithi145 ceph-mon[24853]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:12.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:11 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/1256634877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:33:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:11 smithi145 ceph-mon[29394]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:11 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/1256634877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:33:12.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:11 smithi149 ceph-mon[25685]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:12.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:11 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/1256634877' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:33:13.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:13 smithi149 ceph-mon[25685]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-24T11:33:14.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:13 smithi145 ceph-mon[24853]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-24T11:33:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:13 smithi145 ceph-mon[29394]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-24T11:33:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:15 smithi149 ceph-mon[25685]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:33:16.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:15 smithi145 ceph-mon[24853]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:33:16.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:15 smithi145 ceph-mon[29394]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:33:17.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:16 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:33:17.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:16 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:17.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:33:17.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:16 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:17.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:16 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:33:17.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:16 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:17 smithi149 ceph-mon[25685]: Deploying daemon osd.6 on smithi149 2024-03-24T11:33:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:17 smithi149 ceph-mon[25685]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:33:18.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:17 smithi145 ceph-mon[29394]: Deploying daemon osd.6 on smithi149 2024-03-24T11:33:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:17 smithi145 ceph-mon[29394]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:33:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:17 smithi145 ceph-mon[24853]: Deploying daemon osd.6 on smithi149 2024-03-24T11:33:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:17 smithi145 ceph-mon[24853]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:33:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:19 smithi149 ceph-mon[25685]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:19 smithi145 ceph-mon[24853]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:19 smithi145 ceph-mon[29394]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:21.966 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:21 smithi149 ceph-mon[25685]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:22.193 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:21 smithi145 ceph-mon[24853]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:22.193 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:21 smithi145 ceph-mon[29394]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:23.380 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:23 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:23.380 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:23 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:23.380 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:23 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:23.380 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:23 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:23.380 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:23 smithi149 ceph-mon[25685]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:23.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:23.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:23.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:23.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:23.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[24853]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:23.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:23.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:23.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:23.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:23.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:23 smithi145 ceph-mon[29394]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:25.662 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 6 on host 'smithi149' 2024-03-24T11:33:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:33:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:33:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:25.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:25.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:25 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:26.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:26.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:26.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:33:26.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:26.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:26.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:26.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:25 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:26.316 DEBUG:teuthology.orchestra.run.smithi149:osd.6> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T11:33:26.318 INFO:tasks.cephadm:Deploying osd.7 on smithi149 with /dev/vg_nvme/lv_1... 2024-03-24T11:33:26.318 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:33:27.145 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-mon[25685]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:33:27.145 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-mon[25685]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:33:27.145 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:27.145 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-mon[25685]: from='osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:27.145 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-mon[25685]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:27.145 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-mon[25685]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:27.145 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[34891]: 2024-03-24T11:33:27.136+0000 7fa734bc7700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T11:33:27.146 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:33:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[34891]: 2024-03-24T11:33:27.144+0000 7fa731562700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:33:27.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[24853]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[24853]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[24853]: from='osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[24853]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[24853]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[29394]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[29394]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[29394]: from='osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[29394]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:27.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:27 smithi145 ceph-mon[29394]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:33:28.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:28 smithi149 ceph-mon[25685]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:28.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:28 smithi149 ceph-mon[25685]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:33:28.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:28 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:28 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:28 smithi149 ceph-mon[25685]: osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247] boot 2024-03-24T11:33:28.444 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:28 smithi149 ceph-mon[25685]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:33:28.444 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:28 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[24853]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[24853]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[24853]: osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247] boot 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[24853]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[29394]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[29394]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:28.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[29394]: osd.6 [v2:172.21.15.149:6816/3716387247,v1:172.21.15.149:6817/3716387247] boot 2024-03-24T11:33:28.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[29394]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T11:33:28.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:28 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:33:29.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:29 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:33:29.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:29 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:33:29.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:29 smithi149 ceph-mon[25685]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:33:29.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:29 smithi149 ceph-mon[25685]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:29.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:33:29.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:33:29.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[24853]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:33:29.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[24853]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:29.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:33:29.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:33:29.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[29394]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T11:33:29.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:29 smithi145 ceph-mon[29394]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-03-24T11:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-03-24T11:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123057 s, 852 MB/s 2024-03-24T11:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-03-24T11:33:31.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[24853]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:33:31.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[29394]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:30 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.211 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:30 smithi149 ceph-mon[25685]: osdmap e49: 7 total, 7 up, 7 in 2024-03-24T11:33:31.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:31.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.212 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:30 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:31.235 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi149:vg_nvme/lv_1 2024-03-24T11:33:32.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi149 2024-03-24T11:33:32.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi149 to 1938M 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[29394]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[29394]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[29394]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi149 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi149 to 1938M 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[24853]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[24853]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:31 smithi145 ceph-mon[24853]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:33:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:31 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi149 2024-03-24T11:33:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:31 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi149 to 1938M 2024-03-24T11:33:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:31 smithi149 ceph-mon[25685]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:33:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:31 smithi149 ceph-mon[25685]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:31 smithi149 ceph-mon[25685]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:33:33.958 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:33 smithi149 ceph-mon[25685]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:33.959 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:33:33.959 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:33:33.959 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:33 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:34.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[29394]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:34.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:33:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:33:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[24853]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:33:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:33:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:33 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:34 smithi149 ceph-mon[25685]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:33:35.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:34 smithi145 ceph-mon[24853]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:33:35.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:34 smithi145 ceph-mon[29394]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:33:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[24853]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/2825832480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]: dispatch 2024-03-24T11:33:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]: dispatch 2024-03-24T11:33:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[24853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]': finished 2024-03-24T11:33:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[24853]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:33:36.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[29394]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/2825832480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]: dispatch 2024-03-24T11:33:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]: dispatch 2024-03-24T11:33:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[29394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]': finished 2024-03-24T11:33:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[29394]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:33:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:35 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:35 smithi149 ceph-mon[25685]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:35 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/2825832480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]: dispatch 2024-03-24T11:33:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:35 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]: dispatch 2024-03-24T11:33:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:35 smithi149 ceph-mon[25685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ee0ec17-0116-4f59-9633-c1375ceb5a51"}]': finished 2024-03-24T11:33:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:35 smithi149 ceph-mon[25685]: osdmap e52: 8 total, 7 up, 8 in 2024-03-24T11:33:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:35 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:37.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:36 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/3348996537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:33:37.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:36 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/3348996537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:33:37.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:36 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/3348996537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:33:38.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:37 smithi145 ceph-mon[24853]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:38.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:37 smithi145 ceph-mon[29394]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:37 smithi149 ceph-mon[25685]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:33:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:39 smithi149 ceph-mon[25685]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:33:40.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:39 smithi145 ceph-mon[29394]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:33:40.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:39 smithi145 ceph-mon[24853]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:33:42.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:41 smithi149 ceph-mon[25685]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T11:33:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:41 smithi145 ceph-mon[29394]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T11:33:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:41 smithi145 ceph-mon[24853]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T11:33:43.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:42 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:33:43.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:42 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:42 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:33:43.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:42 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:42 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:33:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:42 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:44.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:43 smithi149 ceph-mon[25685]: Deploying daemon osd.7 on smithi149 2024-03-24T11:33:44.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:43 smithi149 ceph-mon[25685]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:33:44.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:43 smithi145 ceph-mon[29394]: Deploying daemon osd.7 on smithi149 2024-03-24T11:33:44.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:43 smithi145 ceph-mon[29394]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:33:44.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:43 smithi145 ceph-mon[24853]: Deploying daemon osd.7 on smithi149 2024-03-24T11:33:44.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:43 smithi145 ceph-mon[24853]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-24T11:33:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:45 smithi149 ceph-mon[25685]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:46.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:45 smithi145 ceph-mon[29394]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:46.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:45 smithi145 ceph-mon[24853]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:47 smithi149 ceph-mon[25685]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:47 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:48.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[29394]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:48.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[24853]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:47 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:50.011 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:49 smithi149 ceph-mon[25685]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:50.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:49 smithi145 ceph-mon[29394]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:50.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:49 smithi145 ceph-mon[24853]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:50.889 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 7 on host 'smithi149' 2024-03-24T11:33:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:33:51.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:33:51.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:51.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:33:51.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:33:51.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:51.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:51.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:51 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:51.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:33:51.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:51 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:33:51.602 DEBUG:teuthology.orchestra.run.smithi149:osd.7> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T11:33:51.604 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-24T11:33:51.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:33:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:52 smithi149 ceph-mon[25685]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:33:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:52 smithi149 ceph-mon[25685]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:33:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:52 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:52 smithi149 ceph-mon[25685]: from='osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:52 smithi149 ceph-mon[25685]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:52.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:33:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-24T11:33:52.042+0000 7fd87a1d2700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T11:33:52.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:33:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-24T11:33:52.049+0000 7fd871b63700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[24853]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[24853]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[24853]: from='osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[24853]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[29394]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[29394]: osdmap e53: 8 total, 7 up, 8 in 2024-03-24T11:33:52.255 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:52.256 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[29394]: from='osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:52.256 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:52 smithi145 ceph-mon[29394]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898] boot 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[29394]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898] boot 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:53 smithi145 ceph-mon[24853]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:53.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-24T11:33:53.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: osdmap e54: 8 total, 7 up, 8 in 2024-03-24T11:33:53.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: osd.7 [v2:172.21.15.149:6824/439891898,v1:172.21.15.149:6825/439891898] boot 2024-03-24T11:33:53.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T11:33:53.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:33:53.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:53 smithi149 ceph-mon[25685]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:33:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:33:54.147 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711280032,"num_in_osds":8,"osd_in_since":1711280015,"num_remapped_pgs":1} 2024-03-24T11:33:54.147 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:33:54.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[24853]: purged_snaps scrub starts 2024-03-24T11:33:54.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[24853]: purged_snaps scrub ok 2024-03-24T11:33:54.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2463384207' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:33:54.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[24853]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:33:54.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[29394]: purged_snaps scrub starts 2024-03-24T11:33:54.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[29394]: purged_snaps scrub ok 2024-03-24T11:33:54.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2463384207' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:33:54.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:54 smithi145 ceph-mon[29394]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:33:54.369 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:54.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:54 smithi149 ceph-mon[25685]: purged_snaps scrub starts 2024-03-24T11:33:54.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:54 smithi149 ceph-mon[25685]: purged_snaps scrub ok 2024-03-24T11:33:54.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:54 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2463384207' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:33:54.407 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:54 smithi149 ceph-mon[25685]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: Detected new or changed devices on smithi149 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi149 to 1454M 2024-03-24T11:33:55.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.686 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.686 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:33:55.686 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[24853]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:33:55.687 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: Detected new or changed devices on smithi149 2024-03-24T11:33:55.689 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.689 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.689 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.690 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.690 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.690 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi149 to 1454M 2024-03-24T11:33:55.690 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.690 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.690 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:33:55.690 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:55 smithi145 ceph-mon[29394]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: Detected new or changed devices on smithi149 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi149 to 1454M 2024-03-24T11:33:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:33:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: osdmap e57: 8 total, 8 up, 8 in 2024-03-24T11:33:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:55 smithi149 ceph-mon[25685]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:33:56.086 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:33:56.087 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":58,"fsid":"a2bb64be-e9d1-11ee-95cd-87774f69a715","created":"2024-03-24T11:28:58.369383+0000","modified":"2024-03-24T11:33:55.843079+0000","last_up_change":"2024-03-24T11:33:52.831237+0000","last_in_change":"2024-03-24T11:33:35.687172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:31:55.161174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0f61b5c1-0892-46a5-b37d-403044def119","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6803","nonce":488109160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6805","nonce":488109160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6809","nonce":488109160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6807","nonce":488109160}]},"public_addr":"172.21.15.145:6803/488109160","cluster_addr":"172.21.15.145:6805/488109160","heartbeat_back_addr":"172.21.15.145:6809/488109160","heartbeat_front_addr":"172.21.15.145:6807/488109160","state":["exists","up"]},{"osd":1,"uuid":"564e785b-94b7-4f54-a7d1-d65101b260d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6811","nonce":544022099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6813","nonce":544022099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6817","nonce":544022099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6815","nonce":544022099}]},"public_addr":"172.21.15.145:6811/544022099","cluster_addr":"172.21.15.145:6813/544022099","heartbeat_back_addr":"172.21.15.145:6817/544022099","heartbeat_front_addr":"172.21.15.145:6815/544022099","state":["exists","up"]},{"osd":2,"uuid":"d4f6183d-f2de-46bf-ac81-271da6c29463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6819","nonce":1601008660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6821","nonce":1601008660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6825","nonce":1601008660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6823","nonce":1601008660}]},"public_addr":"172.21.15.145:6819/1601008660","cluster_addr":"172.21.15.145:6821/1601008660","heartbeat_back_addr":"172.21.15.145:6825/1601008660","heartbeat_front_addr":"172.21.15.145:6823/1601008660","state":["exists","up"]},{"osd":3,"uuid":"580cea31-9ffe-44e7-bdab-d36551219e6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6827","nonce":3558706490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6829","nonce":3558706490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6833","nonce":3558706490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6831","nonce":3558706490}]},"public_addr":"172.21.15.145:6827/3558706490","cluster_addr":"172.21.15.145:6829/3558706490","heartbeat_back_addr":"172.21.15.145:6833/3558706490","heartbeat_front_addr":"172.21.15.145:6831/3558706490","state":["exists","up"]},{"osd":4,"uuid":"8a6d81d6-ffae-47b5-af8d-b22437386ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6801","nonce":3015006003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6803","nonce":3015006003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6807","nonce":3015006003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6805","nonce":3015006003}]},"public_addr":"172.21.15.149:6801/3015006003","cluster_addr":"172.21.15.149:6803/3015006003","heartbeat_back_addr":"172.21.15.149:6807/3015006003","heartbeat_front_addr":"172.21.15.149:6805/3015006003","state":["exists","up"]},{"osd":5,"uuid":"af3286eb-019d-47b0-9faa-e2d816203030","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6809","nonce":4057913332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6811","nonce":4057913332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6815","nonce":4057913332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6813","nonce":4057913332}]},"public_addr":"172.21.15.149:6809/4057913332","cluster_addr":"172.21.15.149:6811/4057913332","heartbeat_back_addr":"172.21.15.149:6815/4057913332","heartbeat_front_addr":"172.21.15.149:6813/4057913332","state":["exists","up"]},{"osd":6,"uuid":"3f448d3f-b9a0-4ba1-bea9-ddde2fce734a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6817","nonce":3716387247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6819","nonce":3716387247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6823","nonce":3716387247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6821","nonce":3716387247}]},"public_addr":"172.21.15.149:6817/3716387247","cluster_addr":"172.21.15.149:6819/3716387247","heartbeat_back_addr":"172.21.15.149:6823/3716387247","heartbeat_front_addr":"172.21.15.149:6821/3716387247","state":["exists","up"]},{"osd":7,"uuid":"4ee0ec17-0116-4f59-9633-c1375ceb5a51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6825","nonce":439891898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6827","nonce":439891898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6831","nonce":439891898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6829","nonce":439891898}]},"public_addr":"172.21.15.149:6825/439891898","cluster_addr":"172.21.15.149:6827/439891898","heartbeat_back_addr":"172.21.15.149:6831/439891898","heartbeat_front_addr":"172.21.15.149:6829/439891898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:05.953267+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:28.530919+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:51.871195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:32:15.442189+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:32:37.438216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:01.600972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:26.252249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:51.417410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/3876704990":"2024-03-25T11:29:50.784140+0000","172.21.15.145:0/3535593282":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6801/877499941":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6800/877499941":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6801/3212195359":"2024-03-25T11:29:28.268679+0000","172.21.15.145:6800/3212195359":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/1228964895":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/3234105728":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/2888711157":"2024-03-25T11:29:50.784140+0000","172.21.15.145:0/195323836":"2024-03-25T11:29:28.268679+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:33:56.867 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:31:55.161174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-24T11:33:56.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:33:57.082 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:57.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:56 smithi145 ceph-mon[24853]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:33:57.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:56 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4095420612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:33:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:56 smithi145 ceph-mon[29394]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:33:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:56 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4095420612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:33:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:56 smithi149 ceph-mon[25685]: osdmap e58: 8 total, 8 up, 8 in 2024-03-24T11:33:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:56 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4095420612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:33:58.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:57 smithi145 ceph-mon[29394]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:33:58.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:57 smithi145 ceph-mon[29394]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:33:58.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:57 smithi145 ceph-mon[24853]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:33:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:57 smithi145 ceph-mon[24853]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:33:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:57 smithi149 ceph-mon[25685]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:33:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:57 smithi149 ceph-mon[25685]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:33:58.765 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-24T11:33:59.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:58 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2568197297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:33:59.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:58 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2568197297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:33:59.126 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:58 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2568197297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:33:59.357 INFO:tasks.cephadm:Adding prometheus.a on smithi149 2024-03-24T11:33:59.357 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi149=a' 2024-03-24T11:33:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:33:59 smithi149 ceph-mon[25685]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T11:34:00.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:33:59 smithi145 ceph-mon[29394]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T11:34:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:33:59 smithi145 ceph-mon[24853]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T11:34:01.319 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled prometheus update... 2024-03-24T11:34:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:01 smithi149 ceph-mon[25685]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:34:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:34:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:34:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:34:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:34:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:01 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:34:02.009 DEBUG:teuthology.orchestra.run.smithi149:prometheus.a> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T11:34:02.012 INFO:tasks.cephadm:Adding node-exporter.a on smithi145 2024-03-24T11:34:02.012 INFO:tasks.cephadm:Adding node-exporter.b on smithi149 2024-03-24T11:34:02.012 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi145=a;smithi149=b' 2024-03-24T11:34:02.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[24853]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:34:02.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:34:02.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[29394]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' 2024-03-24T11:34:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:01 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:34:02.698 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ignoring --setuser ceph since I am not root 2024-03-24T11:34:02.698 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ignoring --setgroup ceph since I am not root 2024-03-24T11:34:02.698 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-mgr[26874]: -- 172.21.15.149:0/4086783709 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be15a16340 con 0x55be159eec00 2024-03-24T11:34:02.843 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setuser ceph since I am not root 2024-03-24T11:34:02.843 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setgroup ceph since I am not root 2024-03-24T11:34:02.843 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mgr[25092]: -- 172.21.15.145:0/527731788 <== mon.1 v2:172.21.15.145:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a858f6e340 con 0x55a858f46c00 2024-03-24T11:34:02.843 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:02.842+0000 7f7cadcb7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:34:02.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:02.849+0000 7f082cf41000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:34:02.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:02.936+0000 7f082cf41000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:34:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-mon[25685]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-mon[25685]: Saving service prometheus spec with placement smithi149=a;count:1 2024-03-24T11:34:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-mon[25685]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:34:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:02 smithi149 ceph-mon[25685]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:34:03.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:02.932+0000 7f7cadcb7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[24853]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[24853]: Saving service prometheus spec with placement smithi149=a;count:1 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[24853]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[24853]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[29394]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[29394]: Saving service prometheus spec with placement smithi149=a;count:1 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[29394]: from='mgr.14150 172.21.15.145:0/2987158611' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:34:03.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:02 smithi145 ceph-mon[29394]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:34:03.602 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:03.536+0000 7f082cf41000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:34:03.842 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:03.519+0000 7f7cadcb7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:34:03.842 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:03.608+0000 7f7cadcb7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:34:03.842 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:03.691+0000 7f7cadcb7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:34:03.842 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:03.841+0000 7f7cadcb7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:34:03.870 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:03.629+0000 7f082cf41000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:34:03.870 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:03.711+0000 7f082cf41000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:34:03.870 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:03.867+0000 7f082cf41000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:34:04.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:03.922+0000 7f7cadcb7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:34:04.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:04.029+0000 7f7cadcb7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:04.141 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:03.949+0000 7f082cf41000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:34:04.141 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:04.058+0000 7f082cf41000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:04.141 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:04.139+0000 7f082cf41000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:34:04.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:04.301+0000 7f082cf41000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:34:04.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:04.386+0000 7f082cf41000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:34:04.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:04.112+0000 7f7cadcb7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:34:04.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:04.274+0000 7f7cadcb7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:34:04.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:04.359+0000 7f7cadcb7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:34:05.696 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:05.374+0000 7f7cadcb7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:34:05.696 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:05.503+0000 7f7cadcb7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:34:05.696 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:05.695+0000 7f7cadcb7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:34:05.708 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:05.397+0000 7f082cf41000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:34:05.708 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:05.523+0000 7f082cf41000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:34:05.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:05.707+0000 7f082cf41000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:34:05.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:05.792+0000 7f082cf41000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:34:05.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:05.923+0000 7f082cf41000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:06.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:05.783+0000 7f7cadcb7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:34:06.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:05.921+0000 7f7cadcb7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:06.474 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:06.250+0000 7f7cadcb7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:34:06.537 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:06.241+0000 7f082cf41000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:34:06.537 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:06.455+0000 7f082cf41000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:06.538 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:06.536+0000 7f082cf41000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:06.730 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:06.473+0000 7f7cadcb7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:34:06.730 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:06.558+0000 7f7cadcb7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:34:06.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:06.702+0000 7f082cf41000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:34:07.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:06.729+0000 7f7cadcb7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:34:07.891 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:07.603+0000 7f082cf41000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:34:07.971 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:07.671+0000 7f7cadcb7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:34:07.971 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:07.970+0000 7f7cadcb7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:34:08.144 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:07.891+0000 7f082cf41000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:34:08.144 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:07.982+0000 7f082cf41000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:34:08.144 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:08.128+0000 7f082cf41000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:34:08.144 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: [24/Mar/2024:11:34:08] ENGINE Bus STARTING 2024-03-24T11:34:08.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:08.067+0000 7f7cadcb7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:34:08.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:08.218+0000 7f7cadcb7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:34:08.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: CherryPy Checker: 2024-03-24T11:34:08.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: The Application mounted at '' has an empty config. 2024-03-24T11:34:08.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: 2024-03-24T11:34:08.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: [24/Mar/2024:11:34:08] ENGINE Serving on http://:::9283 2024-03-24T11:34:08.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: [24/Mar/2024:11:34:08] ENGINE Bus STARTED 2024-03-24T11:34:09.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: Standby manager daemon x restarted 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: Standby manager daemon x started 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: Active manager daemon y restarted 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: Activating manager daemon y 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[29394]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:34:09.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:34:08] ENGINE Bus STARTING 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: Standby manager daemon x restarted 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: Standby manager daemon x started 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: Active manager daemon y restarted 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: Activating manager daemon y 2024-03-24T11:34:09.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:08 smithi145 ceph-mon[24853]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:34:09.170 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: Standby manager daemon x restarted 2024-03-24T11:34:09.170 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: Standby manager daemon x started 2024-03-24T11:34:09.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:34:09.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:34:09.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:34:09.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/194718271' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:34:09.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: Active manager daemon y restarted 2024-03-24T11:34:09.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: Activating manager daemon y 2024-03-24T11:34:09.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:08 smithi149 ceph-mon[25685]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: CherryPy Checker: 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: The Application mounted at '' has an empty config. 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:34:09] ENGINE Serving on http://:::9283 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:34:09] ENGINE Bus STARTED 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:34:09] ENGINE Bus STARTING 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:34:09] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:34:09.498 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:34:09] ENGINE Bus STARTED 2024-03-24T11:34:09.864 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled node-exporter update... 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: mgrmap e17: y(active, starting, since 0.614581s), standbys: x 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:34:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: Manager daemon y is now available 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: [24/Mar/2024:11:34:09] ENGINE Bus STARTING 2024-03-24T11:34:10.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: mgrmap e17: y(active, starting, since 0.614581s), standbys: x 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:34:10.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: Manager daemon y is now available 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:10.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:34:10.116 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:34:10.116 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: [24/Mar/2024:11:34:09] ENGINE Bus STARTING 2024-03-24T11:34:10.116 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: mgrmap e17: y(active, starting, since 0.614581s), standbys: x 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:34:10.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: Manager daemon y is now available 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:34:10.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: [24/Mar/2024:11:34:09] ENGINE Bus STARTING 2024-03-24T11:34:10.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:10.557 DEBUG:teuthology.orchestra.run.smithi145:node-exporter.a> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T11:34:10.559 DEBUG:teuthology.orchestra.run.smithi149:node-exporter.b> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T11:34:10.561 INFO:tasks.cephadm:Adding alertmanager.a on smithi145 2024-03-24T11:34:10.562 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi145=a' 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: [24/Mar/2024:11:34:09] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: [24/Mar/2024:11:34:09] ENGINE Bus STARTED 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: mgrmap e18: y(active, since 1.62161s), standbys: x 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi145=a;smithi149=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: Saving service node-exporter spec with placement smithi145=a;smithi149=b;count:2 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: [24/Mar/2024:11:34:09] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: [24/Mar/2024:11:34:09] ENGINE Bus STARTED 2024-03-24T11:34:11.097 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: mgrmap e18: y(active, since 1.62161s), standbys: x 2024-03-24T11:34:11.098 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi145=a;smithi149=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:11.098 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:11.098 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: Saving service node-exporter spec with placement smithi145=a;smithi149=b;count:2 2024-03-24T11:34:11.098 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:11.098 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:10 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: [24/Mar/2024:11:34:09] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: [24/Mar/2024:11:34:09] ENGINE Bus STARTED 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: mgrmap e18: y(active, since 1.62161s), standbys: x 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi145=a;smithi149=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: Saving service node-exporter spec with placement smithi145=a;smithi149=b;count:2 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:11.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:10 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:12.149 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:11 smithi149 ceph-mon[25685]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:12.149 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:11 smithi149 ceph-mon[25685]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-24T11:34:12.150 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:12.150 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:11 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:34:12.150 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:11 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:34:12.150 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:12.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[24853]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:12.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[24853]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-24T11:34:12.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[29394]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[29394]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:34:12.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:12.786 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled alertmanager update... 2024-03-24T11:34:13.549 DEBUG:teuthology.orchestra.run.smithi145:alertmanager.a> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T11:34:13.552 INFO:tasks.cephadm:Adding grafana.a on smithi149 2024-03-24T11:34:13.552 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi149=a' 2024-03-24T11:34:13.808 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:13 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.808 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:13 smithi149 ceph-mon[25685]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:13.808 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:13 smithi149 ceph-mon[25685]: Saving service alertmanager spec with placement smithi145=a;count:1 2024-03-24T11:34:13.808 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:13 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.808 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:13 smithi149 ceph-mon[25685]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:13.808 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:13 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[29394]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:13.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[29394]: Saving service alertmanager spec with placement smithi145=a;count:1 2024-03-24T11:34:13.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[29394]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[24853]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[24853]: Saving service alertmanager spec with placement smithi145=a;count:1 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[24853]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:13.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:13 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:15 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:34:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:15 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:34:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:15 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:15 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:15 smithi149 ceph-mon[25685]: Deploying daemon prometheus.a on smithi149 2024-03-24T11:34:15.543 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled grafana update... 2024-03-24T11:34:15.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:34:15.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[24853]: Deploying daemon prometheus.a on smithi149 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:15.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:15 smithi145 ceph-mon[29394]: Deploying daemon prometheus.a on smithi149 2024-03-24T11:34:16.343 DEBUG:teuthology.orchestra.run.smithi149:grafana.a> sudo journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T11:34:16.345 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:34:16.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:34:16.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:16 smithi149 ceph-mon[25685]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:16.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:16 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:16.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:16 smithi145 ceph-mon[29394]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:16.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:16 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:16.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:16 smithi145 ceph-mon[24853]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:16.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:16 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:17.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:17 smithi145 ceph-mon[29394]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:17.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:17 smithi145 ceph-mon[29394]: Saving service grafana spec with placement smithi149=a;count:1 2024-03-24T11:34:17.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:17 smithi145 ceph-mon[24853]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:17.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:17 smithi145 ceph-mon[24853]: Saving service grafana spec with placement smithi149=a;count:1 2024-03-24T11:34:17.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:17 smithi149 ceph-mon[25685]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:34:17.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:17 smithi149 ceph-mon[25685]: Saving service grafana spec with placement smithi149=a;count:1 2024-03-24T11:34:18.316 INFO:teuthology.orchestra.run.smithi145.stdout:[client.0] 2024-03-24T11:34:18.316 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQC6DwBmwVCJEhAAAxAhBR2i9UQ7f7e8I7Tx2A== 2024-03-24T11:34:18.317 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:18 smithi145 ceph-mon[24853]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:18.317 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:18 smithi145 ceph-mon[29394]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:18 smithi149 ceph-mon[25685]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:18.993 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-24T11:34:18.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:34:18.994 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:34:19.036 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:34:19.343 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:19 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/243442382' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:19.344 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:19 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:19.344 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:19 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:34:19.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:19 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/243442382' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:19.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:19 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:19.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:19 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:34:19.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:19 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/243442382' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:19.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:19 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:19.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:19 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:34:20.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:20 smithi145 ceph-mon[29394]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:20.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:20 smithi145 ceph-mon[24853]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:20.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:20 smithi149 ceph-mon[25685]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:22.575 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:22 smithi149 ceph-mon[25685]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:22.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:22 smithi145 ceph-mon[29394]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:22.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:22 smithi145 ceph-mon[24853]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:22.800 INFO:teuthology.orchestra.run.smithi149.stdout:[client.1] 2024-03-24T11:34:22.801 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQC+DwBmrSBkLxAAYpAH04+7hUbpTEB9OEGVyQ== 2024-03-24T11:34:23.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:23 smithi149 ceph-mon[25685]: from='client.? 172.21.15.149:0/2165780863' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:23.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:23 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:23.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:23 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:34:23.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:23 smithi145 ceph-mon[29394]: from='client.? 172.21.15.149:0/2165780863' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:23.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:23 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:23.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:23 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:34:23.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:23 smithi145 ceph-mon[24853]: from='client.? 172.21.15.149:0/2165780863' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:23.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:23 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:34:23.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:23 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:34:24.572 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:24 smithi149 ceph-mon[25685]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:24.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:24 smithi145 ceph-mon[29394]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:24.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:24 smithi145 ceph-mon[24853]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:25.912 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-24T11:34:25.913 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T11:34:25.913 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T11:34:25.958 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:34:25.958 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:34:25.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:34:26.180 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:26.530 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:26 smithi145 ceph-mon[29394]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:26.531 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:26 smithi145 ceph-mon[24853]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:26.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:26 smithi149 ceph-mon[25685]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:26.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:26 smithi149 systemd[1]: Starting Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:34:27.293 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 podman[40798]: 2024-03-24 11:34:26.939160126 +0000 UTC m=+0.018513376 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-24T11:34:27.293 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 podman[40798]: 2024-03-24 11:34:27.066511947 +0000 UTC m=+0.145865184 container create 992150f3bdbba6faae9484941e9e32b206f3912a08843303fdc9df023480cbef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:34:27.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 podman[40798]: 2024-03-24 11:34:27.293692186 +0000 UTC m=+0.373045425 container init 992150f3bdbba6faae9484941e9e32b206f3912a08843303fdc9df023480cbef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 podman[40798]: 2024-03-24 11:34:27.296287926 +0000 UTC m=+0.375641165 container start 992150f3bdbba6faae9484941e9e32b206f3912a08843303fdc9df023480cbef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.319Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.319Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.319Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.319Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi149 (none))" 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.319Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.319Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.320Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.320Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.321Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.324Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:34:27.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.324Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.179µs 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.324Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.324Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.324Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.212µs wal_replay_duration=341.055µs total_replay_duration=395.917µs 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.325Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.325Z caller=main.go:947 level=info msg="TSDB started" 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.325Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.341Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.856048ms db_storage=631ns remote_storage=1.347µs web_handler=252ns query_engine=584ns scrape=367.602µs scrape_sd=16.507µs notify=789ns notify_sd=1.468µs rules=15.306476ms 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:34:27.341Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 bash[40798]: 992150f3bdbba6faae9484941e9e32b206f3912a08843303fdc9df023480cbef 2024-03-24T11:34:27.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:34:27 smithi149 systemd[1]: Started Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:34:27.968 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:34:28.525 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:28 smithi145 ceph-mon[29394]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:28.525 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:28 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:28.525 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:28 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3317875920' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:34:28.525 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:28 smithi145 ceph-mon[24853]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:28.525 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:28 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:28.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:28 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3317875920' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:34:28.549 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":19,"active_gid":14442,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2169988987},{"type":"v1","addr":"172.21.15.145:6801","nonce":2169988987}]},"active_addr":"172.21.15.145:6801/2169988987","active_change":"2024-03-24T11:34:08.232356+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24335,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.145:8443/","prometheus":"http://172.21.15.145:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":1874048018}]},{"addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":2743538076}]},{"addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":2920519761}]},{"addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":2864908609}]}]}} 2024-03-24T11:34:28.552 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:34:28.553 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:34:28.553 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:34:28.689 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:28 smithi149 ceph-mon[25685]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:28.690 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:28 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:28.690 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:28 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3317875920' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:34:28.790 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:28.861 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:28 smithi145 bash[44186]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-24T11:34:29.245 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:29 smithi145 bash[44186]: Getting image source signatures 2024-03-24T11:34:29.245 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:29 smithi145 bash[44186]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-24T11:34:29.245 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:29 smithi145 bash[44186]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-24T11:34:29.245 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:29 smithi145 bash[44186]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-24T11:34:29.596 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:29 smithi149 ceph-mon[25685]: Deploying daemon node-exporter.a on smithi145 2024-03-24T11:34:29.596 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:29 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:29.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:29 smithi145 ceph-mon[29394]: Deploying daemon node-exporter.a on smithi145 2024-03-24T11:34:29.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:29 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:29.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:29 smithi145 ceph-mon[24853]: Deploying daemon node-exporter.a on smithi145 2024-03-24T11:34:29.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:29 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:30.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:30 smithi145 ceph-mon[29394]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:30.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:30 smithi145 ceph-mon[24853]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:30.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:30 smithi149 ceph-mon[25685]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:31.256 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:30 smithi145 bash[44186]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-24T11:34:31.584 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:31 smithi145 bash[44186]: Writing manifest to image destination 2024-03-24T11:34:31.708 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:34:31.708 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":60,"fsid":"a2bb64be-e9d1-11ee-95cd-87774f69a715","created":"2024-03-24T11:28:58.369383+0000","modified":"2024-03-24T11:34:08.231986+0000","last_up_change":"2024-03-24T11:33:52.831237+0000","last_in_change":"2024-03-24T11:33:35.687172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:31:55.161174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0f61b5c1-0892-46a5-b37d-403044def119","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6803","nonce":488109160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6805","nonce":488109160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6809","nonce":488109160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6807","nonce":488109160}]},"public_addr":"172.21.15.145:6803/488109160","cluster_addr":"172.21.15.145:6805/488109160","heartbeat_back_addr":"172.21.15.145:6809/488109160","heartbeat_front_addr":"172.21.15.145:6807/488109160","state":["exists","up"]},{"osd":1,"uuid":"564e785b-94b7-4f54-a7d1-d65101b260d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6811","nonce":544022099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6813","nonce":544022099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6817","nonce":544022099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6815","nonce":544022099}]},"public_addr":"172.21.15.145:6811/544022099","cluster_addr":"172.21.15.145:6813/544022099","heartbeat_back_addr":"172.21.15.145:6817/544022099","heartbeat_front_addr":"172.21.15.145:6815/544022099","state":["exists","up"]},{"osd":2,"uuid":"d4f6183d-f2de-46bf-ac81-271da6c29463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6819","nonce":1601008660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6821","nonce":1601008660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6825","nonce":1601008660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6823","nonce":1601008660}]},"public_addr":"172.21.15.145:6819/1601008660","cluster_addr":"172.21.15.145:6821/1601008660","heartbeat_back_addr":"172.21.15.145:6825/1601008660","heartbeat_front_addr":"172.21.15.145:6823/1601008660","state":["exists","up"]},{"osd":3,"uuid":"580cea31-9ffe-44e7-bdab-d36551219e6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6827","nonce":3558706490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6829","nonce":3558706490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6833","nonce":3558706490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6831","nonce":3558706490}]},"public_addr":"172.21.15.145:6827/3558706490","cluster_addr":"172.21.15.145:6829/3558706490","heartbeat_back_addr":"172.21.15.145:6833/3558706490","heartbeat_front_addr":"172.21.15.145:6831/3558706490","state":["exists","up"]},{"osd":4,"uuid":"8a6d81d6-ffae-47b5-af8d-b22437386ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6801","nonce":3015006003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6803","nonce":3015006003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6807","nonce":3015006003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6805","nonce":3015006003}]},"public_addr":"172.21.15.149:6801/3015006003","cluster_addr":"172.21.15.149:6803/3015006003","heartbeat_back_addr":"172.21.15.149:6807/3015006003","heartbeat_front_addr":"172.21.15.149:6805/3015006003","state":["exists","up"]},{"osd":5,"uuid":"af3286eb-019d-47b0-9faa-e2d816203030","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6809","nonce":4057913332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6811","nonce":4057913332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6815","nonce":4057913332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6813","nonce":4057913332}]},"public_addr":"172.21.15.149:6809/4057913332","cluster_addr":"172.21.15.149:6811/4057913332","heartbeat_back_addr":"172.21.15.149:6815/4057913332","heartbeat_front_addr":"172.21.15.149:6813/4057913332","state":["exists","up"]},{"osd":6,"uuid":"3f448d3f-b9a0-4ba1-bea9-ddde2fce734a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6817","nonce":3716387247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6819","nonce":3716387247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6823","nonce":3716387247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6821","nonce":3716387247}]},"public_addr":"172.21.15.149:6817/3716387247","cluster_addr":"172.21.15.149:6819/3716387247","heartbeat_back_addr":"172.21.15.149:6823/3716387247","heartbeat_front_addr":"172.21.15.149:6821/3716387247","state":["exists","up"]},{"osd":7,"uuid":"4ee0ec17-0116-4f59-9633-c1375ceb5a51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6825","nonce":439891898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6827","nonce":439891898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6831","nonce":439891898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6829","nonce":439891898}]},"public_addr":"172.21.15.149:6825/439891898","cluster_addr":"172.21.15.149:6827/439891898","heartbeat_back_addr":"172.21.15.149:6831/439891898","heartbeat_front_addr":"172.21.15.149:6829/439891898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:05.953267+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:28.530919+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:51.871195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:32:15.442189+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:32:37.438216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:01.600972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:26.252249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:51.417410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/791882194":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/2872389050":"2024-03-25T11:34:08.231953+0000","172.21.15.145:6801/1914003978":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/2182110881":"2024-03-25T11:34:08.231953+0000","172.21.15.145:6800/1914003978":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/3876704990":"2024-03-25T11:29:50.784140+0000","172.21.15.145:0/2398354259":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/3535593282":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6801/877499941":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6800/877499941":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6801/3212195359":"2024-03-25T11:29:28.268679+0000","172.21.15.145:6800/3212195359":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/1228964895":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/3234105728":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/2888711157":"2024-03-25T11:29:50.784140+0000","172.21.15.145:0/195323836":"2024-03-25T11:29:28.268679+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:34:31.849 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:31 smithi145 podman[44186]: 2024-03-24 11:34:31.670695326 +0000 UTC m=+3.081015862 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-24T11:34:31.849 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:31 smithi145 podman[44186]: 2024-03-24 11:34:31.778265656 +0000 UTC m=+3.188586235 container create 7399120d8cc1c529a5128bdddc8e9fcc85e8e34a5af55bcea872c5920fc89335 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:34:32.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-mon[24853]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:32.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3338188243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:34:32.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:32.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-mon[29394]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:32.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3338188243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:34:32.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 podman[44186]: 2024-03-24 11:34:32.148086772 +0000 UTC m=+3.558407336 container init 7399120d8cc1c529a5128bdddc8e9fcc85e8e34a5af55bcea872c5920fc89335 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 podman[44186]: 2024-03-24 11:34:32.150952908 +0000 UTC m=+3.561273455 container start 7399120d8cc1c529a5128bdddc8e9fcc85e8e34a5af55bcea872c5920fc89335 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.154Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.154Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=arp 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-24T11:34:32.613 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=edac 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-24T11:34:32.614 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=os 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-24T11:34:32.615 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=stat 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=time 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=uname 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.155Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-24T11:34:32.616 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[44354]: ts=2024-03-24T11:34:32.156Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-24T11:34:32.617 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 bash[44186]: 7399120d8cc1c529a5128bdddc8e9fcc85e8e34a5af55bcea872c5920fc89335 2024-03-24T11:34:32.617 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:34:32 smithi145 systemd[1]: Started Ceph node-exporter.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:34:32.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:32 smithi149 ceph-mon[25685]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:32.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:32 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3338188243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:34:32.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:32 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:32.791 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:34:32.791 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:34:33.011 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:33.345 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:33 smithi149 bash[41051]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-24T11:34:33.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:33 smithi145 ceph-mon[24853]: Deploying daemon node-exporter.b on smithi149 2024-03-24T11:34:33.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:33 smithi145 ceph-mon[29394]: Deploying daemon node-exporter.b on smithi149 2024-03-24T11:34:33.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:33 smithi149 ceph-mon[25685]: Deploying daemon node-exporter.b on smithi149 2024-03-24T11:34:34.225 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:33 smithi149 bash[41051]: Getting image source signatures 2024-03-24T11:34:34.225 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:33 smithi149 bash[41051]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-24T11:34:34.225 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:33 smithi149 bash[41051]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-24T11:34:34.225 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:33 smithi149 bash[41051]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-24T11:34:34.342 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:34:34] "GET /metrics HTTP/1.1" 200 191524 "" "Prometheus/2.33.4" 2024-03-24T11:34:34.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:34 smithi145 ceph-mon[24853]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:34.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:34 smithi145 ceph-mon[29394]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:34.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:34 smithi149 ceph-mon[25685]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:34.913 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:34:34.913 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":60,"fsid":"a2bb64be-e9d1-11ee-95cd-87774f69a715","created":"2024-03-24T11:28:58.369383+0000","modified":"2024-03-24T11:34:08.231986+0000","last_up_change":"2024-03-24T11:33:52.831237+0000","last_in_change":"2024-03-24T11:33:35.687172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:31:55.161174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0f61b5c1-0892-46a5-b37d-403044def119","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6803","nonce":488109160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6805","nonce":488109160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6809","nonce":488109160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":488109160},{"type":"v1","addr":"172.21.15.145:6807","nonce":488109160}]},"public_addr":"172.21.15.145:6803/488109160","cluster_addr":"172.21.15.145:6805/488109160","heartbeat_back_addr":"172.21.15.145:6809/488109160","heartbeat_front_addr":"172.21.15.145:6807/488109160","state":["exists","up"]},{"osd":1,"uuid":"564e785b-94b7-4f54-a7d1-d65101b260d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6811","nonce":544022099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6813","nonce":544022099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6817","nonce":544022099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":544022099},{"type":"v1","addr":"172.21.15.145:6815","nonce":544022099}]},"public_addr":"172.21.15.145:6811/544022099","cluster_addr":"172.21.15.145:6813/544022099","heartbeat_back_addr":"172.21.15.145:6817/544022099","heartbeat_front_addr":"172.21.15.145:6815/544022099","state":["exists","up"]},{"osd":2,"uuid":"d4f6183d-f2de-46bf-ac81-271da6c29463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6819","nonce":1601008660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6821","nonce":1601008660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6825","nonce":1601008660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1601008660},{"type":"v1","addr":"172.21.15.145:6823","nonce":1601008660}]},"public_addr":"172.21.15.145:6819/1601008660","cluster_addr":"172.21.15.145:6821/1601008660","heartbeat_back_addr":"172.21.15.145:6825/1601008660","heartbeat_front_addr":"172.21.15.145:6823/1601008660","state":["exists","up"]},{"osd":3,"uuid":"580cea31-9ffe-44e7-bdab-d36551219e6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6827","nonce":3558706490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6829","nonce":3558706490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6833","nonce":3558706490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3558706490},{"type":"v1","addr":"172.21.15.145:6831","nonce":3558706490}]},"public_addr":"172.21.15.145:6827/3558706490","cluster_addr":"172.21.15.145:6829/3558706490","heartbeat_back_addr":"172.21.15.145:6833/3558706490","heartbeat_front_addr":"172.21.15.145:6831/3558706490","state":["exists","up"]},{"osd":4,"uuid":"8a6d81d6-ffae-47b5-af8d-b22437386ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6801","nonce":3015006003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6803","nonce":3015006003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6807","nonce":3015006003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":3015006003},{"type":"v1","addr":"172.21.15.149:6805","nonce":3015006003}]},"public_addr":"172.21.15.149:6801/3015006003","cluster_addr":"172.21.15.149:6803/3015006003","heartbeat_back_addr":"172.21.15.149:6807/3015006003","heartbeat_front_addr":"172.21.15.149:6805/3015006003","state":["exists","up"]},{"osd":5,"uuid":"af3286eb-019d-47b0-9faa-e2d816203030","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6809","nonce":4057913332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6811","nonce":4057913332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6815","nonce":4057913332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":4057913332},{"type":"v1","addr":"172.21.15.149:6813","nonce":4057913332}]},"public_addr":"172.21.15.149:6809/4057913332","cluster_addr":"172.21.15.149:6811/4057913332","heartbeat_back_addr":"172.21.15.149:6815/4057913332","heartbeat_front_addr":"172.21.15.149:6813/4057913332","state":["exists","up"]},{"osd":6,"uuid":"3f448d3f-b9a0-4ba1-bea9-ddde2fce734a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6817","nonce":3716387247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6819","nonce":3716387247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6823","nonce":3716387247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":3716387247},{"type":"v1","addr":"172.21.15.149:6821","nonce":3716387247}]},"public_addr":"172.21.15.149:6817/3716387247","cluster_addr":"172.21.15.149:6819/3716387247","heartbeat_back_addr":"172.21.15.149:6823/3716387247","heartbeat_front_addr":"172.21.15.149:6821/3716387247","state":["exists","up"]},{"osd":7,"uuid":"4ee0ec17-0116-4f59-9633-c1375ceb5a51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6825","nonce":439891898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6827","nonce":439891898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6831","nonce":439891898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":439891898},{"type":"v1","addr":"172.21.15.149:6829","nonce":439891898}]},"public_addr":"172.21.15.149:6825/439891898","cluster_addr":"172.21.15.149:6827/439891898","heartbeat_back_addr":"172.21.15.149:6831/439891898","heartbeat_front_addr":"172.21.15.149:6829/439891898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:05.953267+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:28.530919+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:31:51.871195+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:32:15.442189+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:32:37.438216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:01.600972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:26.252249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:33:51.417410+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/791882194":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/2872389050":"2024-03-25T11:34:08.231953+0000","172.21.15.145:6801/1914003978":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/2182110881":"2024-03-25T11:34:08.231953+0000","172.21.15.145:6800/1914003978":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/3876704990":"2024-03-25T11:29:50.784140+0000","172.21.15.145:0/2398354259":"2024-03-25T11:34:08.231953+0000","172.21.15.145:0/3535593282":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6801/877499941":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6800/877499941":"2024-03-25T11:29:50.784140+0000","172.21.15.145:6801/3212195359":"2024-03-25T11:29:28.268679+0000","172.21.15.145:6800/3212195359":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/1228964895":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/3234105728":"2024-03-25T11:29:28.268679+0000","172.21.15.145:0/2888711157":"2024-03-25T11:29:50.784140+0000","172.21.15.145:0/195323836":"2024-03-25T11:29:28.268679+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:34:35.600 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:35 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2198488463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:34:35.600 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:35 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2198488463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:34:35.625 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:34:35.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:34:35.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:34:35.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T11:34:35.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T11:34:35.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T11:34:35.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T11:34:35.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T11:34:35.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:35 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2198488463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:34:36.068 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.068 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.073 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.076 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.080 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.080 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.083 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.089 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-mon[25685]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:36.475 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 bash[41051]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-24T11:34:36.475 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 bash[41051]: Writing manifest to image destination 2024-03-24T11:34:36.497 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:36 smithi145 ceph-mon[29394]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:36.498 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:36 smithi145 ceph-mon[24853]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:36.905 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 podman[41051]: 2024-03-24 11:34:36.588009301 +0000 UTC m=+3.506366674 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-24T11:34:36.905 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 podman[41051]: 2024-03-24 11:34:36.678344444 +0000 UTC m=+3.596701855 container create edb4947ca62a6b7151d17a1ef2f75bed13f440405471da77a36a9b6064101fab (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 podman[41051]: 2024-03-24 11:34:36.905409706 +0000 UTC m=+3.823767071 container init edb4947ca62a6b7151d17a1ef2f75bed13f440405471da77a36a9b6064101fab (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 podman[41051]: 2024-03-24 11:34:36.908617145 +0000 UTC m=+3.826974513 container start edb4947ca62a6b7151d17a1ef2f75bed13f440405471da77a36a9b6064101fab (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.912Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.912Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=arp 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-24T11:34:37.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=edac 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-24T11:34:37.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=os 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-24T11:34:37.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=stat 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-24T11:34:37.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=time 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=uname 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[41100]: ts=2024-03-24T11:34:36.913Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 bash[41051]: edb4947ca62a6b7151d17a1ef2f75bed13f440405471da77a36a9b6064101fab 2024-03-24T11:34:37.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:34:36 smithi149 systemd[1]: Started Ceph node-exporter.b for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:34:38.051 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:34:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:38 smithi145 ceph-mon[24853]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:38 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:38 smithi145 ceph-mon[24853]: Deploying daemon alertmanager.a on smithi145 2024-03-24T11:34:38.276 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:38 smithi145 ceph-mon[29394]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:38.276 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:38 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:38.276 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:38 smithi145 ceph-mon[29394]: Deploying daemon alertmanager.a on smithi145 2024-03-24T11:34:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:38 smithi149 ceph-mon[25685]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:38 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:38 smithi149 ceph-mon[25685]: Deploying daemon alertmanager.a on smithi145 2024-03-24T11:34:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:39 smithi149 ceph-mon[25685]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:39 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:40.187 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:39 smithi145 ceph-mon[29394]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:40.188 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:39 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:40.188 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:39 smithi145 ceph-mon[24853]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:40.188 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:39 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:34:42.194 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:41 smithi145 ceph-mon[24853]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:42.194 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:41 smithi145 ceph-mon[29394]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:41 smithi149 ceph-mon[25685]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:44.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:43 smithi145 ceph-mon[24853]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:44.030 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:43 smithi145 ceph-mon[29394]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:43 smithi149 ceph-mon[25685]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:44.327 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:34:44] "GET /metrics HTTP/1.1" 200 191507 "" "Prometheus/2.33.4" 2024-03-24T11:34:44.971 INFO:teuthology.orchestra.run.smithi145.stdout:133143986202 2024-03-24T11:34:44.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T11:34:45.910 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:45 smithi145 ceph-mon[24853]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:45 smithi149 ceph-mon[25685]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:46.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:45 smithi145 ceph-mon[29394]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:46.495 INFO:teuthology.orchestra.run.smithi145.stdout:111669149727 2024-03-24T11:34:46.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T11:34:47.791 INFO:teuthology.orchestra.run.smithi145.stdout:34359738413 2024-03-24T11:34:47.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:34:48.017 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:47 smithi145 ceph-mon[29394]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:48.017 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:47 smithi145 ceph-mon[24853]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:47 smithi149 ceph-mon[25685]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:48.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:34:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:34:49.020 INFO:teuthology.orchestra.run.smithi145.stdout:167503724566 2024-03-24T11:34:49.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T11:34:49.633 INFO:teuthology.orchestra.run.smithi145.stdout:55834574889 2024-03-24T11:34:49.633 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:34:50.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:49 smithi149 ceph-mon[25685]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:50.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:49 smithi145 ceph-mon[24853]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:50.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:49 smithi145 ceph-mon[29394]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:51.196 INFO:teuthology.orchestra.run.smithi145.stdout:236223201292 2024-03-24T11:34:51.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T11:34:51.945 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:51.946 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:51.946 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:51.946 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:52.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:51 smithi145 ceph-mon[24853]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:52.098 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:51 smithi145 ceph-mon[29394]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:51 smithi149 ceph-mon[25685]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:52.591 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:52.591 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:54.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:53 smithi145 ceph-mon[24853]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:54.031 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:53 smithi145 ceph-mon[29394]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:54.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:53 smithi149 ceph-mon[25685]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:54.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:34:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:34:54] "GET /metrics HTTP/1.1" 200 191496 "" "Prometheus/2.33.4" 2024-03-24T11:34:55.484 INFO:teuthology.orchestra.run.smithi145.stdout:201863462930 2024-03-24T11:34:55.484 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T11:34:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:55 smithi149 ceph-mon[25685]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:56.360 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:55 smithi145 ceph-mon[24853]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:56.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:55 smithi145 ceph-mon[29394]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:58.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:57 smithi145 ceph-mon[29394]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:57 smithi145 ceph-mon[24853]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:57 smithi149 ceph-mon[25685]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:34:58.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:34:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:34:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:35:00.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:59 smithi149 ceph-mon[25685]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:00.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:34:59 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/928607211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:35:00.259 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:59 smithi145 ceph-mon[24853]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:00.259 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:34:59 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/928607211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:35:00.260 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:59 smithi145 ceph-mon[29394]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:00.260 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:34:59 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/928607211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:35:01.464 INFO:teuthology.orchestra.run.smithi145.stdout:236223201295 2024-03-24T11:35:01.868 INFO:teuthology.orchestra.run.smithi145.stdout:111669149731 2024-03-24T11:35:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:01 smithi149 ceph-mon[25685]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:01 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2402170282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:35:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:01 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4169615555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:35:02.310 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:01 smithi145 ceph-mon[29394]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:02.311 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:01 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2402170282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:35:02.311 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:01 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4169615555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:35:02.311 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:01 smithi145 ceph-mon[24853]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:02.311 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:01 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2402170282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:35:02.311 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:01 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4169615555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:35:02.516 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201292 got 236223201295 for osd.7 2024-03-24T11:35:02.516 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:02.645 INFO:teuthology.orchestra.run.smithi145.stdout:34359738415 2024-03-24T11:35:03.169 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738415 for osd.0 2024-03-24T11:35:03.169 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:03.788 INFO:teuthology.orchestra.run.smithi145.stdout:77309411367 2024-03-24T11:35:03.789 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:35:04.021 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:03 smithi145 ceph-mon[24853]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:04.021 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:03 smithi145 ceph-mon[29394]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:03 smithi149 ceph-mon[25685]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:04.341 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:35:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:04] "GET /metrics HTTP/1.1" 200 191496 "" "Prometheus/2.33.4" 2024-03-24T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stdout:133143986206 2024-03-24T11:35:04.960 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:04 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1506072963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:35:04.961 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:04 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1506072963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:35:05.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:04 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1506072963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:35:05.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986202 got 133143986206 for osd.4 2024-03-24T11:35:05.635 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:05.912 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:06.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:05 smithi145 ceph-mon[29394]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:06.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:05 smithi145 ceph-mon[24853]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:05 smithi149 ceph-mon[25685]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:08.011 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:07 smithi145 ceph-mon[29394]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:08.011 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:07 smithi145 ceph-mon[24853]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:08.103 INFO:teuthology.orchestra.run.smithi145.stdout:55834574894 2024-03-24T11:35:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:07 smithi149 ceph-mon[25685]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:08.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:35:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:35:08.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149731 for osd.3 2024-03-24T11:35:08.964 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:09.129 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:08 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/747778536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:35:09.233 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:08 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/747778536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:35:09.233 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:08 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/747778536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:35:09.336 INFO:teuthology.orchestra.run.smithi145.stdout:167503724570 2024-03-24T11:35:10.052 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[24853]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:10.052 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:35:10.052 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:35:10.053 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3940368529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:35:10.053 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[29394]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:10.053 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:35:10.053 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:35:10.053 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:09 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3940368529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:35:10.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724566 got 167503724570 for osd.5 2024-03-24T11:35:10.205 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:09 smithi149 ceph-mon[25685]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:35:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:09 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:35:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:09 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3940368529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:35:11.272 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574894 for osd.1 2024-03-24T11:35:11.272 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:11.660 INFO:teuthology.orchestra.run.smithi145.stdout:201863462934 2024-03-24T11:35:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:11 smithi149 ceph-mon[25685]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:12.096 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:12.096 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:12.096 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:12.096 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:12.096 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[29394]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:12.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:12.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:35:12.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:12.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:12.097 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:11 smithi145 ceph-mon[24853]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:12.397 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462930 got 201863462934 for osd.6 2024-03-24T11:35:12.397 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:12.643 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:12 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[46968]: level=info ts=2024-03-24T11:35:12.500Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000525269s 2024-03-24T11:35:12.948 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:12 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4174812883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:35:12.949 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:12 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4174812883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:35:12.971 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:12 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4174812883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:35:13.330 INFO:teuthology.orchestra.run.smithi145.stdout:77309411369 2024-03-24T11:35:14.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:14] "GET /metrics HTTP/1.1" 200 191497 "" "Prometheus/2.33.4" 2024-03-24T11:35:14.274 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-03-24T11:35:14.275 DEBUG:teuthology.parallel:result is None 2024-03-24T11:35:14.275 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:35:14.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:35:14.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:14 smithi149 ceph-mon[25685]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:14.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:14.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:14 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3976809685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:35:14.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:14.606 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:14.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[24853]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:14.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:14.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3976809685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:35:14.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:14.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[29394]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:14.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:14.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3976809685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:35:14.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:16.296 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:35:16.297 INFO:teuthology.orchestra.run.smithi145.stderr:dumped all 2024-03-24T11:35:16.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:16 smithi149 ceph-mon[25685]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:16.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:16 smithi145 ceph-mon[29394]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:16.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:16 smithi145 ceph-mon[24853]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:16.979 INFO:teuthology.orchestra.run.smithi145.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-24T11:35:14.862468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48832,"kb_used_data":6144,"kb_used_omap":0,"kb_used_meta":42624,"kb_avail":749912384,"statfs":{"total":767960285184,"available":767910281216,"internally_reserved":0,"allocated":6291456,"data_stored":3510070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43646976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001857"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.956096+0000","last_change":"2024-03-24T11:33:56.852813+0000","last_active":"2024-03-24T11:34:08.956096+0000","last_peered":"2024-03-24T11:34:08.956096+0000","last_clean":"2024-03-24T11:34:08.956096+0000","last_became_active":"2024-03-24T11:33:56.852381+0000","last_became_peered":"2024-03-24T11:33:56.852381+0000","last_unstale":"2024-03-24T11:34:08.956096+0000","last_undegraded":"2024-03-24T11:34:08.956096+0000","last_fullsized":"2024-03-24T11:34:08.956096+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:55.809465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:55.809465+0000","last_clean_scrub_stamp":"2024-03-24T11:31:55.809465+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:53:45.829826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.49399999999999999}]},{"osd":1,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48399999999999999}]},{"osd":3,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.437},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.432}]},{"osd":4,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33400000000000002}]},{"osd":5,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45300000000000001}]},{"osd":6,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.376}]}]},{"osd":6,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.67000000000000004}]},{"osd":1,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.65800000000000003}]},{"osd":2,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.626},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.71299999999999997}]},{"osd":3,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.70299999999999996}]},{"osd":4,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.38900000000000001}]},{"osd":5,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.51400000000000001}]},{"osd":7,"last update":"Sun Mar 24 11:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36499999999999999,"15min":0.35299999999999998},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55700000000000005,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.33800000000000002},"min":{"1min":0.309,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.377}]},{"osd":2,"last update":"Sun Mar 24 11:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60999999999999999,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.34100000000000003}]},{"osd":3,"last update":"Sun Mar 24 11:34:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45400000000000001,"15min":0.45500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58999999999999997,"5min":0.62,"15min":0.62},"last":0.433}]},{"osd":4,"last update":"Sun Mar 24 11:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.63300000000000001}]},{"osd":5,"last update":"Sun Mar 24 11:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.44600000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56799999999999995}]},{"osd":6,"last update":"Sun Mar 24 11:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.72699999999999998}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 11:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.35199999999999998,"15min":0.33900000000000002},"min":{"1min":0.26100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34399999999999997}]},{"osd":2,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.378,"15min":0.37},"min":{"1min":0.25600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57399999999999995,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.377},{"interface":"front","average":{"1min":0.376,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.33000000000000002}]},{"osd":3,"last update":"Sun Mar 24 11:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45400000000000001,"15min":0.45500000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42999999999999999,"15min":0.433},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36499999999999999}]},{"osd":4,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.40799999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.753,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53100000000000003}]},{"osd":5,"last update":"Sun Mar 24 11:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76100000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54100000000000004}]},{"osd":6,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.55300000000000005}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6212,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":585728,"data_stored":240049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.373},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.47699999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.27100000000000002}]},{"osd":1,"last update":"Sun Mar 24 11:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.44500000000000001,"15min":0.44800000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51200000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.32000000000000001}]},{"osd":3,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.44500000000000001,"15min":0.44900000000000001},"min":{"1min":0.28699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.52500000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.30399999999999999}]},{"osd":4,"last update":"Sun Mar 24 11:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.40200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.60399999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80500000000000005,"5min":0.879,"15min":0.879},"last":0.50900000000000001}]},{"osd":5,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.39200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64500000000000002}]},{"osd":6,"last update":"Sun Mar 24 11:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54300000000000004}]},{"osd":7,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.71799999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.42299999999999999,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.38},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.437,"15min":0.442},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.37}]},{"osd":1,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.47299999999999998,"15min":0.47599999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.32800000000000001}]},{"osd":4,"last update":"Sun Mar 24 11:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.75,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49099999999999999}]},{"osd":5,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60299999999999998,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.751,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52600000000000002}]},{"osd":6,"last update":"Sun Mar 24 11:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.69099999999999995,"5min":0.873,"15min":0.873},"last":0.38},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51400000000000001,"15min":0.51900000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68500000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71799999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72999999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.5}]},{"osd":2,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68799999999999994,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.53300000000000003},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54500000000000004}]},{"osd":3,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.624,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.432},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.40699999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.66700000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41899999999999998}]},{"osd":5,"last update":"Sun Mar 24 11:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.40200000000000002,"15min":0.39100000000000001},"min":{"1min":0.29899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.42899999999999999,"15min":0.41599999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.52400000000000002}]},{"osd":6,"last update":"Sun Mar 24 11:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59799999999999998}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.48099999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70099999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.38100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.67200000000000004,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.63300000000000001}]},{"osd":1,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63700000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.57199999999999995}]},{"osd":2,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.36699999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64500000000000002,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65600000000000003}]},{"osd":3,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40400000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73799999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.69499999999999995}]},{"osd":4,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.42899999999999999,"15min":0.41899999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39600000000000002}]},{"osd":6,"last update":"Sun Mar 24 11:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.436}]},{"osd":7,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.495}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:35:16.980 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:35:17.301 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:17.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:17 smithi145 ceph-mon[24853]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:17.333 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:17 smithi145 ceph-mon[29394]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:17.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:17 smithi149 ceph-mon[25685]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:18.197 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:35:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:35:18.283 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:18 smithi145 ceph-mon[29394]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:18.283 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:18 smithi145 ceph-mon[24853]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:18 smithi149 ceph-mon[25685]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:19.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:35:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:35:19.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[29394]: Deploying daemon grafana.a on smithi149 2024-03-24T11:35:19.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:19.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.614 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.614 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:35:19.614 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:35:19.614 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.614 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:19 smithi145 ceph-mon[24853]: Deploying daemon grafana.a on smithi149 2024-03-24T11:35:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:35:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:35:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:19 smithi149 ceph-mon[25685]: Deploying daemon grafana.a on smithi149 2024-03-24T11:35:20.371 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:35:20.372 INFO:teuthology.orchestra.run.smithi145.stderr:dumped all 2024-03-24T11:35:20.738 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:20 smithi145 ceph-mon[24853]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:20.739 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:20 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[46968]: level=info ts=2024-03-24T11:35:20.504Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004091538s 2024-03-24T11:35:20.739 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:20 smithi145 ceph-mon[29394]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:20 smithi149 ceph-mon[25685]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:35:21.005 INFO:teuthology.orchestra.run.smithi145.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-24T11:35:18.863228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48832,"kb_used_data":6144,"kb_used_omap":0,"kb_used_meta":42624,"kb_avail":749912384,"statfs":{"total":767960285184,"available":767910281216,"internally_reserved":0,"allocated":6291456,"data_stored":3510070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43646976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002079"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.956096+0000","last_change":"2024-03-24T11:33:56.852813+0000","last_active":"2024-03-24T11:34:08.956096+0000","last_peered":"2024-03-24T11:34:08.956096+0000","last_clean":"2024-03-24T11:34:08.956096+0000","last_became_active":"2024-03-24T11:33:56.852381+0000","last_became_peered":"2024-03-24T11:33:56.852381+0000","last_unstale":"2024-03-24T11:34:08.956096+0000","last_undegraded":"2024-03-24T11:34:08.956096+0000","last_fullsized":"2024-03-24T11:34:08.956096+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:55.809465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:55.809465+0000","last_clean_scrub_stamp":"2024-03-24T11:31:55.809465+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:53:45.829826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.49399999999999999}]},{"osd":1,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48399999999999999}]},{"osd":3,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.437},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.432}]},{"osd":4,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33400000000000002}]},{"osd":5,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.45300000000000001}]},{"osd":6,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.376}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.499}]},{"osd":1,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51800000000000002}]},{"osd":2,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53900000000000003}]},{"osd":3,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55700000000000005}]},{"osd":4,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.61599999999999999}]},{"osd":5,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.624},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.67900000000000005}]},{"osd":7,"last update":"Sun Mar 24 11:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.64800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36499999999999999,"15min":0.35299999999999998},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55700000000000005,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.33800000000000002},"min":{"1min":0.309,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.377}]},{"osd":2,"last update":"Sun Mar 24 11:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60999999999999999,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.34100000000000003}]},{"osd":3,"last update":"Sun Mar 24 11:34:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45400000000000001,"15min":0.45500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58999999999999997,"5min":0.62,"15min":0.62},"last":0.433}]},{"osd":4,"last update":"Sun Mar 24 11:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.63300000000000001}]},{"osd":5,"last update":"Sun Mar 24 11:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.44600000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56799999999999995}]},{"osd":6,"last update":"Sun Mar 24 11:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.72699999999999998}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 11:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.35199999999999998,"15min":0.33900000000000002},"min":{"1min":0.26100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34399999999999997}]},{"osd":2,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.378,"15min":0.37},"min":{"1min":0.25600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57399999999999995,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.377},{"interface":"front","average":{"1min":0.376,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.33000000000000002}]},{"osd":3,"last update":"Sun Mar 24 11:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45400000000000001,"15min":0.45500000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42999999999999999,"15min":0.433},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36499999999999999}]},{"osd":4,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.40799999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.753,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53100000000000003}]},{"osd":5,"last update":"Sun Mar 24 11:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76100000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54100000000000004}]},{"osd":6,"last update":"Sun Mar 24 11:34:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.55300000000000005}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6212,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":585728,"data_stored":240049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.373},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.47699999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.38}]},{"osd":1,"last update":"Sun Mar 24 11:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.44500000000000001,"15min":0.44800000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51200000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41499999999999998}]},{"osd":3,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.44500000000000001,"15min":0.44900000000000001},"min":{"1min":0.28699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.373},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.52500000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.44400000000000001}]},{"osd":4,"last update":"Sun Mar 24 11:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.40200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.437},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.60399999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80500000000000005,"5min":0.879,"15min":0.879},"last":0.61799999999999999}]},{"osd":5,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.39200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.502}]},{"osd":6,"last update":"Sun Mar 24 11:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54400000000000004}]},{"osd":7,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.58999999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.42299999999999999,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.38},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.437,"15min":0.442},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.37}]},{"osd":1,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Mar 24 11:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.47299999999999998,"15min":0.47599999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.32800000000000001}]},{"osd":4,"last update":"Sun Mar 24 11:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.75,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49099999999999999}]},{"osd":5,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60299999999999998,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.751,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52600000000000002}]},{"osd":6,"last update":"Sun Mar 24 11:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.69099999999999995,"5min":0.873,"15min":0.873},"last":0.38},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51400000000000001,"15min":0.51900000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68500000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71799999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72999999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.5}]},{"osd":2,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68799999999999994,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.53300000000000003},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54500000000000004}]},{"osd":3,"last update":"Sun Mar 24 11:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.624,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.432},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.40699999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.66700000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41899999999999998}]},{"osd":5,"last update":"Sun Mar 24 11:35:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.40200000000000002,"15min":0.39100000000000001},"min":{"1min":0.29899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.42899999999999999,"15min":0.41599999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.52400000000000002}]},{"osd":6,"last update":"Sun Mar 24 11:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59799999999999998}]},{"osd":7,"last update":"Sun Mar 24 11:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.48099999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70099999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.38100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.67200000000000004,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.41999999999999998}]},{"osd":1,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63700000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.36299999999999999}]},{"osd":2,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.36699999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.64500000000000002,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.442},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.379}]},{"osd":3,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40400000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73799999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65300000000000002}]},{"osd":4,"last update":"Sun Mar 24 11:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.42899999999999999,"15min":0.41899999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.63400000000000001}]},{"osd":6,"last update":"Sun Mar 24 11:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.67900000000000005}]},{"osd":7,"last update":"Sun Mar 24 11:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:35:21.007 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:35:21.007 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:35:21.007 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:35:21.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:35:21.226 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:21.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:21 smithi145 ceph-mon[29394]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:21.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:21 smithi145 ceph-mon[24853]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:21.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:21 smithi149 ceph-mon[25685]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:22.955 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-24T11:35:22.955 INFO:teuthology.orchestra.run.smithi145.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:35:23.471 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:35:23.471 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:35:23.471 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:35:23.481 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-24T11:35:23.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-03-24T11:35:24.104 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:23 smithi145 ceph-mon[24853]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:24.104 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:23 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1079069394' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:35:24.104 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:35:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:24] "GET /metrics HTTP/1.1" 200 191497 "" "Prometheus/2.33.4" 2024-03-24T11:35:24.105 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:23 smithi145 ceph-mon[29394]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:24.105 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:23 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1079069394' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:35:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:23 smithi149 ceph-mon[25685]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:23 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1079069394' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:35:25.868 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:35:25.877 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-24T11:35:25.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-24T11:35:26.475 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:26 smithi145 ceph-mon[29394]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:26.476 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:26 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1620639177' entity='client.admin' 2024-03-24T11:35:26.476 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:26 smithi145 ceph-mon[24853]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:26.476 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:26 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1620639177' entity='client.admin' 2024-03-24T11:35:26.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:26 smithi149 ceph-mon[25685]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:26.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:26 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1620639177' entity='client.admin' 2024-03-24T11:35:28.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:35:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:35:28.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:28 smithi145 ceph-mon[29394]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:28.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:28 smithi145 ceph-mon[24853]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:28.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:28 smithi149 ceph-mon[25685]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:29.348 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:35:29.348 INFO:teuthology.orchestra.run.smithi145.stdout: "id": "385f405a-84be-44f5-b770-f9f86fe2bcbc", 2024-03-24T11:35:29.348 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "r", 2024-03-24T11:35:29.348 INFO:teuthology.orchestra.run.smithi145.stdout: "current_period": "46bd9094-2ae7-413a-a910-25b96d89c949", 2024-03-24T11:35:29.348 INFO:teuthology.orchestra.run.smithi145.stdout: "epoch": 1 2024-03-24T11:35:29.348 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:35:29.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:29 smithi145 ceph-mon[24853]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:35:29.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:29 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2903257936' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:35:29.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:29 smithi145 ceph-mon[29394]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:35:29.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:29 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2903257936' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:35:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:29 smithi149 ceph-mon[25685]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:35:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:29 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2903257936' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:35:29.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-24T11:35:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:30 smithi149 ceph-mon[25685]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:30 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2903257936' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:35:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:30 smithi149 ceph-mon[25685]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:35:30.597 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:30 smithi145 ceph-mon[29394]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:30.597 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:30 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2903257936' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:35:30.598 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:30 smithi145 ceph-mon[29394]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:35:30.598 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:30 smithi145 ceph-mon[24853]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:35:30.598 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:30 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2903257936' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:35:30.598 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:30 smithi145 ceph-mon[24853]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:35:31.577 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-mon[25685]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:35:31.577 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 podman[41862]: 2024-03-24 11:35:31.446386776 +0000 UTC m=+0.018291245 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-24T11:35:31.577 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 podman[41862]: 2024-03-24 11:35:31.577046787 +0000 UTC m=+0.148951241 container create 1ea96d14b839a799485bdd84db2d44a594635a334028f3d00383c303b13e91de (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, version=8.5, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, name=ubi8, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc.) 2024-03-24T11:35:31.595 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:31 smithi145 ceph-mon[29394]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:35:31.595 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:31 smithi145 ceph-mon[24853]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:35:31.612 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:35:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "id": "7e4ac2c9-fc43-4824-a01c-b24f28afb079", 2024-03-24T11:35:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default", 2024-03-24T11:35:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "api_name": "default", 2024-03-24T11:35:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "is_master": "true", 2024-03-24T11:35:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "endpoints": [], 2024-03-24T11:35:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "hostnames": [], 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "hostnames_s3website": [], 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "master_zone": "", 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "zones": [], 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "placement_targets": [], 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "default_placement": "", 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "realm_id": "385f405a-84be-44f5-b770-f9f86fe2bcbc", 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "sync_policy": { 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "groups": [] 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:31.613 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:35:31.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 podman[41862]: 2024-03-24 11:35:31.804355624 +0000 UTC m=+0.376260084 container init 1ea96d14b839a799485bdd84db2d44a594635a334028f3d00383c303b13e91de (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, vcs-type=git, maintainer=Paul Cuzner , version=8.5, io.buildah.version=1.24.2, io.openshift.expose-services=, io.openshift.tags=base rhel8, release=236.1648460182, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-24T11:35:31.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 podman[41862]: 2024-03-24 11:35:31.807229921 +0000 UTC m=+0.379134376 container start 1ea96d14b839a799485bdd84db2d44a594635a334028f3d00383c303b13e91de (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, vcs-type=git, io.openshift.tags=base rhel8, version=8.5) 2024-03-24T11:35:32.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-24T11:35:32.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-24T11:35:32.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-24T11:35:32.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="App mode production" logger=settings 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T11:35:32.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-24T11:35:32.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-03-24T11:35:32.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-03-24T11:35:32.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-03-24T11:35:32.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-03-24T11:35:32.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 bash[41862]: 1ea96d14b839a799485bdd84db2d44a594635a334028f3d00383c303b13e91de 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 systemd[1]: Started Ceph grafana.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-03-24T11:35:32.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-03-24T11:35:32.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-03-24T11:35:32.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T11:35:32.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-24T11:35:32.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-24T11:35:32.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-24T11:35:32.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-24T11:35:32.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-24T11:35:32.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-24T11:35:32.100 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-24T11:35:32.336 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-mon[25685]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T11:35:32.336 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:32.336 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:32.336 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-24T11:35:32.337 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T11:35:32.338 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-24T11:35:32.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-24T11:35:32.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-24T11:35:32.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-24T11:35:32.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-24T11:35:32.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-24T11:35:32.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T11:35:32.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T11:35:32.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T11:35:32.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-24T11:35:32.345 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-24T11:35:32.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-24T11:35:32.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-24T11:35:32.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-24T11:35:32.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-24T11:35:32.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-24T11:35:32.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T11:35:32.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-24T11:35:32.352 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-24T11:35:32.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-24T11:35:32.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-24T11:35:32.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-24T11:35:32.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-24T11:35:32.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T11:35:32.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-24T11:35:32.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[29394]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T11:35:32.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:32.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:32.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:32.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[24853]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T11:35:32.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:32.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:32.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:32 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:32.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-24T11:35:32.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-24T11:35:32.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-24T11:35:32.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=513.438918ms 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-24T11:35:32.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-24T11:35:32.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-24T11:35:32.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T11:35:32.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-24T11:35:32.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T11:35:32.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:35:32 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:35:32+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-24T11:35:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:33 smithi145 ceph-mon[24853]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T11:35:34.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:35:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:34] "GET /metrics HTTP/1.1" 200 191497 "" "Prometheus/2.33.4" 2024-03-24T11:35:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:33 smithi145 ceph-mon[29394]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T11:35:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:33 smithi149 ceph-mon[25685]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout: "id": "0307e4a0-19f2-4498-adf1-023a3cf115f6", 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "z", 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout: "domain_root": "z.rgw.meta:root", 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout: "control_pool": "z.rgw.control", 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-24T11:35:34.586 INFO:teuthology.orchestra.run.smithi145.stdout: "log_pool": "z.rgw.log", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "otp_pool": "z.rgw.otp", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "system_key": { 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "access_key": "", 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "secret_key": "" 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: "placement_pools": [ 2024-03-24T11:35:34.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "key": "default-placement", 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "val": { 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "storage_classes": { 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "STANDARD": { 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "index_type": 0 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "realm_id": "385f405a-84be-44f5-b770-f9f86fe2bcbc", 2024-03-24T11:35:34.588 INFO:teuthology.orchestra.run.smithi145.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-24T11:35:34.589 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:35:35.147 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:34 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:35.148 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:34 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:35.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-24T11:35:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:34 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:35 smithi145 ceph-mon[24853]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-03-24T11:35:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:35 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:35 smithi145 ceph-mon[29394]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-03-24T11:35:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:35 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:35 smithi149 ceph-mon[25685]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-03-24T11:35:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:35 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:38.063 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:35:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:35:38.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[24853]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[24853]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[24853]: Reconfiguring daemon alertmanager.a on smithi145 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[29394]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[29394]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:35:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:38 smithi145 ceph-mon[29394]: Reconfiguring daemon alertmanager.a on smithi145 2024-03-24T11:35:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:38 smithi149 ceph-mon[25685]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-24T11:35:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:38 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:38 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:38 smithi149 ceph-mon[25685]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:35:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:38 smithi149 ceph-mon[25685]: Reconfiguring daemon alertmanager.a on smithi145 2024-03-24T11:35:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:39 smithi149 ceph-mon[25685]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:35:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:39 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:40.243 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:39 smithi145 ceph-mon[24853]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:35:40.243 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:39 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:40.244 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:39 smithi145 ceph-mon[29394]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-24T11:35:40.244 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:39 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:40.540 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:40 smithi145 systemd[1]: Stopping Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:35:40.541 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:40 smithi145 bash[49298]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-24T11:35:40.832 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:40 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[46968]: level=info ts=2024-03-24T11:35:40.540Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:35:40.832 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:40 smithi145 podman[49309]: 2024-03-24 11:35:40.570825758 +0000 UTC m=+0.152317963 container died a3c63dfb4d152afac40ef793605497071926984e35fc419ba386f50be7a89ac8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:41.111 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:40 smithi145 podman[49309]: 2024-03-24 11:35:40.914901635 +0000 UTC m=+0.496393849 container cleanup a3c63dfb4d152afac40ef793605497071926984e35fc419ba386f50be7a89ac8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:41.111 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:40 smithi145 bash[49309]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:35:41.111 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:40 smithi145 bash[49337]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-24T11:35:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-mon[29394]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:35:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1332574960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:35:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:35:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-mon[24853]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:35:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1332574960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:35:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:35:41.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:41 smithi149 ceph-mon[25685]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:35:41.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:41 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1332574960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:35:41.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:41 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:35:41.576 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 podman[49323]: 2024-03-24 11:35:41.257943069 +0000 UTC m=+0.686631975 container remove a3c63dfb4d152afac40ef793605497071926984e35fc419ba386f50be7a89ac8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:41.576 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 podman[49323]: 2024-03-24 11:35:41.358876453 +0000 UTC m=+0.787565359 volume remove b44e2a548bbf7f237b6a891fe4d382e26c0c8da5d371f1fa34f68aa98fad8264 2024-03-24T11:35:41.576 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:35:41.576 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 systemd[1]: Stopped Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:35:41.576 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 systemd[1]: Starting Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:35:41.849 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 podman[49407]: 2024-03-24 11:35:41.576165583 +0000 UTC m=+0.109568439 volume create 1a5be6e0324a9120481fbaa25af585279a8623f7a6f42a5f0598c276f2dde5a0 2024-03-24T11:35:41.850 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 podman[49407]: 2024-03-24 11:35:41.485113767 +0000 UTC m=+0.018516591 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-24T11:35:41.850 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 podman[49407]: 2024-03-24 11:35:41.659927995 +0000 UTC m=+0.193330809 container create b0ee65c327d9c3ea9990ced7d93234b2baf1f1a92484483fbb513cd34fe5b9a5 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-mon[29394]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-03-24T11:35:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-24T11:35:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-mon[29394]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:35:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-mon[24853]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-03-24T11:35:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-24T11:35:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-mon[24853]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:35:42.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 podman[49407]: 2024-03-24 11:35:41.962776937 +0000 UTC m=+0.496179758 container init b0ee65c327d9c3ea9990ced7d93234b2baf1f1a92484483fbb513cd34fe5b9a5 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:42.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 podman[49407]: 2024-03-24 11:35:41.965389553 +0000 UTC m=+0.498792375 container start b0ee65c327d9c3ea9990ced7d93234b2baf1f1a92484483fbb513cd34fe5b9a5 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:42.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:41.975Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-24T11:35:42.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:41.975Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-24T11:35:42.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:41.976Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.145 port=9094 2024-03-24T11:35:42.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:41.977Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:35:42.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:42.002Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:35:42.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:42.002Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:35:42.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:42.004Z caller=main.go:518 msg=Listening address=:9093 2024-03-24T11:35:42.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:42.004Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-24T11:35:42.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:42 smithi145 bash[49407]: b0ee65c327d9c3ea9990ced7d93234b2baf1f1a92484483fbb513cd34fe5b9a5 2024-03-24T11:35:42.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:42 smithi145 systemd[1]: Started Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:35:42.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:42 smithi149 ceph-mon[25685]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 819 B/s wr, 3 op/s 2024-03-24T11:35:42.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:42 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-24T11:35:42.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:42 smithi149 ceph-mon[25685]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:35:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:35:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1332574960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:35:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[29394]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1332574960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:35:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-24T11:35:43.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-mon[24853]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1332574960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-24T11:35:43.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:43 smithi149 ceph-mon[25685]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:35:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:44 smithi149 ceph-mon[25685]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-03-24T11:35:44.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:35:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:44] "GET /metrics HTTP/1.1" 200 193143 "" "Prometheus/2.33.4" 2024-03-24T11:35:44.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:43.978Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001015628s 2024-03-24T11:35:44.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:44 smithi145 ceph-mon[29394]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-03-24T11:35:44.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:44 smithi145 ceph-mon[24853]: pgmap v56: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-03-24T11:35:45.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:45 smithi145 ceph-mon[24853]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:35:45.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:45 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:35:45.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:45 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:35:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:45 smithi145 ceph-mon[29394]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:35:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:45 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:35:45.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:45 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:35:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:45 smithi149 ceph-mon[25685]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:35:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:45 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:35:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:45 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:35:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:46 smithi149 ceph-mon[25685]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:35:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:46 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:35:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:46 smithi149 ceph-mon[25685]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:35:46.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:46 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:35:46.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:46 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:35:46.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[29394]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:35:46.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[29394]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[24853]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[24853]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:35:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:46 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:35:47.183 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:35:47.183 INFO:teuthology.orchestra.run.smithi145.stdout: "id": "c9a87139-042f-4581-9906-fc87a1b8939d", 2024-03-24T11:35:47.183 INFO:teuthology.orchestra.run.smithi145.stdout: "epoch": 1, 2024-03-24T11:35:47.183 INFO:teuthology.orchestra.run.smithi145.stdout: "predecessor_uuid": "46bd9094-2ae7-413a-a910-25b96d89c949", 2024-03-24T11:35:47.183 INFO:teuthology.orchestra.run.smithi145.stdout: "sync_status": [], 2024-03-24T11:35:47.183 INFO:teuthology.orchestra.run.smithi145.stdout: "period_map": { 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "id": "c9a87139-042f-4581-9906-fc87a1b8939d", 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "zonegroups": [ 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "id": "7e4ac2c9-fc43-4824-a01c-b24f28afb079", 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default", 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "api_name": "default", 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "is_master": "true", 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "endpoints": [], 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "hostnames": [], 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "hostnames_s3website": [], 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "master_zone": "0307e4a0-19f2-4498-adf1-023a3cf115f6", 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: "zones": [ 2024-03-24T11:35:47.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "id": "0307e4a0-19f2-4498-adf1-023a3cf115f6", 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "z", 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "endpoints": [], 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "log_meta": "false", 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "log_data": "false", 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "bucket_index_max_shards": 11, 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "read_only": "false", 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "tier_type": "", 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "sync_from_all": "true", 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "sync_from": [], 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "redirect_zone": "" 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:35:47.185 INFO:teuthology.orchestra.run.smithi145.stdout: "placement_targets": [ 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default-placement", 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "storage_classes": [ 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "STANDARD" 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "default_placement": "default-placement", 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "realm_id": "385f405a-84be-44f5-b770-f9f86fe2bcbc", 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "sync_policy": { 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: "groups": [] 2024-03-24T11:35:47.186 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: "short_zone_ids": [ 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: "key": "0307e4a0-19f2-4498-adf1-023a3cf115f6", 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 1043388710 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: "master_zonegroup": "7e4ac2c9-fc43-4824-a01c-b24f28afb079", 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: "master_zone": "0307e4a0-19f2-4498-adf1-023a3cf115f6", 2024-03-24T11:35:47.187 INFO:teuthology.orchestra.run.smithi145.stdout: "period_config": { 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "bucket_quota": { 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": false, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "check_on_raw": false, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "max_size": -1, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "max_size_kb": 0, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "max_objects": -1 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "user_quota": { 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": false, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "check_on_raw": false, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "max_size": -1, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "max_size_kb": 0, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "max_objects": -1 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:47.188 INFO:teuthology.orchestra.run.smithi145.stdout: "user_ratelimit": { 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_read_ops": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_write_ops": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_read_bytes": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_write_bytes": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": false 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "bucket_ratelimit": { 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_read_ops": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_write_ops": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_read_bytes": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_write_bytes": 0, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": false 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "anonymous_ratelimit": { 2024-03-24T11:35:47.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max_read_ops": 0, 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: "max_write_ops": 0, 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: "max_read_bytes": 0, 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: "max_write_bytes": 0, 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": false 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: "realm_id": "385f405a-84be-44f5-b770-f9f86fe2bcbc", 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: "realm_name": "r", 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout: "realm_epoch": 2 2024-03-24T11:35:47.190 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:35:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:47 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:35:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:47 smithi149 ceph-mon[25685]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:35:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:47 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-24T11:35:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:47 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-24T11:35:47.533 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:35:47.533 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[24853]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:35:47.533 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-24T11:35:47.533 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-24T11:35:47.534 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:35:47.534 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[29394]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:35:47.534 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1927063648' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-24T11:35:47.534 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:47 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-24T11:35:47.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-24T11:35:48.133 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:35:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:35:48.230 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:48 smithi145 ceph-mon[24853]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:35:48.230 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:48 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-24T11:35:48.231 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:48 smithi145 ceph-mon[24853]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:35:48.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:48 smithi145 ceph-mon[29394]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:35:48.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:48 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-24T11:35:48.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:48 smithi145 ceph-mon[29394]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:35:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:48 smithi149 ceph-mon[25685]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-24T11:35:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:48 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-24T11:35:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:48 smithi149 ceph-mon[25685]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:35:49.718 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled rgw.foo update... 2024-03-24T11:35:50.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-mon[25685]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 214 B/s rd, 429 B/s wr, 1 op/s 2024-03-24T11:35:50.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:50.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 systemd[1]: Stopping Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:35:50.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 bash[42478]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-24T11:35:50.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:50 smithi145 ceph-mon[29394]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 214 B/s rd, 429 B/s wr, 1 op/s 2024-03-24T11:35:50.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:50 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:50.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:50 smithi145 ceph-mon[24853]: pgmap v64: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 214 B/s rd, 429 B/s wr, 1 op/s 2024-03-24T11:35:50.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:50 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:50.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch apply rgw smpl' 2024-03-24T11:35:50.561 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:35:50.561 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:35:50.561 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:35:50.561 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-24T11:35:50.561 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:35:50.561 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-24T11:35:50.562 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:35:50.562 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-24T11:35:50.562 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:35:50.562 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:35:50.562 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-24T11:35:50.562 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[40814]: ts=2024-03-24T11:35:50.252Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-24T11:35:50.562 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 podman[42491]: 2024-03-24 11:35:50.263091132 +0000 UTC m=+0.146460339 container died 992150f3bdbba6faae9484941e9e32b206f3912a08843303fdc9df023480cbef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:50.894 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 podman[42491]: 2024-03-24 11:35:50.702193763 +0000 UTC m=+0.585562960 container cleanup 992150f3bdbba6faae9484941e9e32b206f3912a08843303fdc9df023480cbef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:50.894 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 bash[42491]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:35:51.145 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-mon[25685]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:51.145 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 podman[42504]: 2024-03-24 11:35:50.894748613 +0000 UTC m=+0.635155643 container remove 992150f3bdbba6faae9484941e9e32b206f3912a08843303fdc9df023480cbef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:51.145 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 bash[42518]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-24T11:35:51.145 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T11:35:51.145 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 systemd[1]: Stopped Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:35:51.145 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:50 smithi149 systemd[1]: Starting Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:35:51.145 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 podman[42586]: 2024-03-24 11:35:51.040204226 +0000 UTC m=+0.019810726 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-24T11:35:51.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:51 smithi145 ceph-mon[24853]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:51.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:51 smithi145 ceph-mon[24853]: Saving service rgw.foo spec with placement count:2 2024-03-24T11:35:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:51 smithi145 ceph-mon[29394]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:51 smithi145 ceph-mon[29394]: Saving service rgw.foo spec with placement count:2 2024-03-24T11:35:51.414 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-mon[25685]: Saving service rgw.foo spec with placement count:2 2024-03-24T11:35:51.418 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 podman[42586]: 2024-03-24 11:35:51.162507669 +0000 UTC m=+0.142114136 container create b35d6bc7db17ca524016cc019c00c8df83fa4798f0cbe44c0dc8fcb8f294c82c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:51.673 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 podman[42586]: 2024-03-24 11:35:51.414753152 +0000 UTC m=+0.394359632 container init b35d6bc7db17ca524016cc019c00c8df83fa4798f0cbe44c0dc8fcb8f294c82c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 podman[42586]: 2024-03-24 11:35:51.417384035 +0000 UTC m=+0.396990513 container start b35d6bc7db17ca524016cc019c00c8df83fa4798f0cbe44c0dc8fcb8f294c82c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.441Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.441Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.441Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.441Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi149 (none))" 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.441Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.441Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.442Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.443Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.444Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-24T11:35:51.674 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.446Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.446Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.657µs 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.446Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 bash[42586]: b35d6bc7db17ca524016cc019c00c8df83fa4798f0cbe44c0dc8fcb8f294c82c 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 systemd[1]: Started Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.654Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.655Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.655Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.77µs wal_replay_duration=208.786377ms total_replay_duration=208.825099ms 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.657Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.657Z caller=main.go:947 level=info msg="TSDB started" 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.657Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.673Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.946547ms db_storage=874ns remote_storage=2.15µs web_handler=419ns query_engine=720ns scrape=397.765µs scrape_sd=24.733µs notify=21.129µs notify_sd=7.989µs rules=15.118759ms 2024-03-24T11:35:51.675 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:35:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:35:51.673Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-24T11:35:52.336 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled rgw.smpl update... 2024-03-24T11:35:52.337 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:35:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:35:51.980Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002380515s 2024-03-24T11:35:52.337 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:35:52.337 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.337 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:35:52.337 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.145:9093"}]: dispatch 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:35:52.338 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.145:9093"}]: dispatch 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:52.339 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:52 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:52.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-24T11:35:52.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.145:9093"}]: dispatch 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:52.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:52 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.145:9093"}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: Saving service rgw.smpl spec with placement count:2 2024-03-24T11:35:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:53 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:53.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:35:53.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.145:9093"}]: dispatch 2024-03-24T11:35:53.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:35:53.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-03-24T11:35:53.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:35:53.510 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: Saving service rgw.smpl spec with placement count:2 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.145:9093"}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: Saving service rgw.smpl spec with placement count:2 2024-03-24T11:35:53.511 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:53 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:53.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph osd pool create foo' 2024-03-24T11:35:54.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:54 smithi145 ceph-mon[29394]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 922 B/s wr, 2 op/s 2024-03-24T11:35:54.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:54 smithi145 ceph-mon[24853]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 922 B/s wr, 2 op/s 2024-03-24T11:35:54.677 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:54 smithi149 ceph-mon[25685]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 922 B/s wr, 2 op/s 2024-03-24T11:35:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:55 smithi149 ceph-mon[25685]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 818 B/s wr, 2 op/s 2024-03-24T11:35:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:55 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:55 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3894152609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-24T11:35:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:55 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-24T11:35:56.225 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[29394]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 818 B/s wr, 2 op/s 2024-03-24T11:35:56.225 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:56.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3894152609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-24T11:35:56.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-24T11:35:56.226 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[24853]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 818 B/s wr, 2 op/s 2024-03-24T11:35:56.226 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:56.226 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3894152609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-24T11:35:56.226 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:55 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-24T11:35:56.361 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'foo' created 2024-03-24T11:35:57.328 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'rbd pool init foo' 2024-03-24T11:35:57.571 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-24T11:35:57.571 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:35:57.571 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.571 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.572 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.573 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.573 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.573 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:35:57.573 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:35:57.573 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.573 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:57 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:57.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-24T11:35:57.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:35:57.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:35:57.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:35:57.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:35:57.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:58.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:35:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:35:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:35:58.562 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[29394]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 734 B/s wr, 1 op/s 2024-03-24T11:35:58.563 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[29394]: Saving service rgw.foo spec with placement count:2 2024-03-24T11:35:58.563 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[29394]: Deploying daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:35:58.563 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[29394]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:35:58.563 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[24853]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 734 B/s wr, 1 op/s 2024-03-24T11:35:58.563 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[24853]: Saving service rgw.foo spec with placement count:2 2024-03-24T11:35:58.563 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[24853]: Deploying daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:35:58.563 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:58 smithi145 ceph-mon[24853]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:35:58.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:58 smithi149 ceph-mon[25685]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 734 B/s wr, 1 op/s 2024-03-24T11:35:58.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:58 smithi149 ceph-mon[25685]: Saving service rgw.foo spec with placement count:2 2024-03-24T11:35:58.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:58 smithi149 ceph-mon[25685]: Deploying daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:35:58.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:58 smithi149 ceph-mon[25685]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:35:59.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:35:59 smithi149 ceph-mon[25685]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:35:59.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:35:59 smithi145 ceph-mon[24853]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:35:59.770 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:35:59 smithi145 ceph-mon[29394]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:36:00.450 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[24853]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:00.450 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:00.450 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:00.450 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:00.450 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:00.450 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:00.451 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[29394]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:00.451 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:00.451 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:00.451 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:00.451 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:00.451 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:00 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:00.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:00 smithi149 ceph-mon[25685]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:00.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:00 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:00.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:00 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:00.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:00 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:00.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:00 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:00.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:00 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:01.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:01 smithi149 ceph-mon[25685]: Deploying daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:36:01.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:01 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2094753047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-24T11:36:01.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:01 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-24T11:36:01.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:01 smithi145 ceph-mon[24853]: Deploying daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:36:01.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:01 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2094753047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-24T11:36:01.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:01 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-24T11:36:01.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:01 smithi145 ceph-mon[29394]: Deploying daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:36:01.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:01 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2094753047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-24T11:36:01.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:01 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-24T11:36:02.465 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:02 smithi149 ceph-mon[25685]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:02.465 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:02 smithi149 ceph-mon[25685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-24T11:36:02.465 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:02 smithi149 ceph-mon[25685]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:36:02.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:02 smithi145 ceph-mon[29394]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:02.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:02 smithi145 ceph-mon[29394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-24T11:36:02.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:02 smithi145 ceph-mon[29394]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:36:02.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:02 smithi145 ceph-mon[24853]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:02.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:02 smithi145 ceph-mon[24853]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-24T11:36:02.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:02 smithi145 ceph-mon[24853]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:36:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:03 smithi149 ceph-mon[25685]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:36:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:03 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:03 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:03 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:03 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:03 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:03 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[24853]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[29394]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:36:03.781 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.782 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.782 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:03.782 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:03.782 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:03.782 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:03 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:04.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:04] "GET /metrics HTTP/1.1" 200 196856 "" "Prometheus/2.33.4" 2024-03-24T11:36:04.242 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-24T11:36:04.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[24853]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:04.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[24853]: Saving service rgw.smpl spec with placement count:2 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[24853]: Deploying daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[24853]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[29394]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[29394]: Saving service rgw.smpl spec with placement count:2 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[29394]: Deploying daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[29394]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:36:04.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:04 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:04.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:04 smithi149 ceph-mon[25685]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:04.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:04 smithi149 ceph-mon[25685]: Saving service rgw.smpl spec with placement count:2 2024-03-24T11:36:04.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:04 smithi149 ceph-mon[25685]: Deploying daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:36:04.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:04 smithi149 ceph-mon[25685]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:36:04.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:04 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:06.509 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:06 smithi145 ceph-mon[24853]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 6.0 KiB/s wr, 354 op/s 2024-03-24T11:36:06.509 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:06 smithi145 ceph-mon[29394]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 6.0 KiB/s wr, 354 op/s 2024-03-24T11:36:06.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:06 smithi149 ceph-mon[25685]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 6.0 KiB/s wr, 354 op/s 2024-03-24T11:36:06.988 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled iscsi.foo update... 2024-03-24T11:36:07.974 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:36:08.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'sleep 120' 2024-03-24T11:36:08.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 6.0 KiB/s wr, 354 op/s 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: Saving service iscsi.foo spec with placement count:1 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[24853]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:36:08.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 6.0 KiB/s wr, 354 op/s 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: Saving service iscsi.foo spec with placement count:1 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:08.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:07 smithi145 ceph-mon[29394]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:36:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 6.0 KiB/s wr, 354 op/s 2024-03-24T11:36:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:36:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: Saving service iscsi.foo spec with placement count:1 2024-03-24T11:36:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:36:08.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:36:08.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:08.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:08.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:07 smithi149 ceph-mon[25685]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:36:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:08 smithi149 ceph-mon[25685]: Deploying daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:36:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-03-24T11:36:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T11:36:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-24T11:36:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:36:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:36:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:36:09.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[24853]: Deploying daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:36:09.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[29394]: Deploying daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:36:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:36:10.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[29394]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 5.5 KiB/s wr, 327 op/s 2024-03-24T11:36:10.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-03-24T11:36:10.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[29394]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[24853]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 5.5 KiB/s wr, 327 op/s 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:36:10.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:10 smithi145 ceph-mon[24853]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:36:10.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:10 smithi149 ceph-mon[25685]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 5.5 KiB/s wr, 327 op/s 2024-03-24T11:36:10.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:10 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-03-24T11:36:10.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:10 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T11:36:10.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:10 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-24T11:36:10.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:10 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:36:10.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:10 smithi149 ceph-mon[25685]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:36:11.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[24853]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:36:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:11.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:11.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[29394]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:36:11.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:11.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:11.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:11 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:11 smithi149 ceph-mon[25685]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:36:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:11 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:12.266 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:12 smithi149 ceph-mon[25685]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-24T11:36:12.290 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:12 smithi145 ceph-mon[24853]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-24T11:36:12.290 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:12 smithi145 ceph-mon[29394]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-24T11:36:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:13 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:13 smithi149 ceph-mon[25685]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-24T11:36:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:13 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:14.240 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:13 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:14.240 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:13 smithi145 ceph-mon[29394]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-24T11:36:14.240 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:13 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:14.240 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:14] "GET /metrics HTTP/1.1" 200 210660 "" "Prometheus/2.33.4" 2024-03-24T11:36:14.240 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:13 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:14.240 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:13 smithi145 ceph-mon[24853]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-24T11:36:14.240 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:13 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: Checking dashboard <-> RGW credentials 2024-03-24T11:36:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: Checking pool "foo" exists for service iscsi.foo 2024-03-24T11:36:15.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:36:15.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:36:15.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:15.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:14 smithi149 ceph-mon[25685]: Deploying daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:36:15.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: Checking dashboard <-> RGW credentials 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: Checking pool "foo" exists for service iscsi.foo 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[29394]: Deploying daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:36:15.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: Checking dashboard <-> RGW credentials 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: Checking pool "foo" exists for service iscsi.foo 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:15.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:14 smithi145 ceph-mon[24853]: Deploying daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:36:16.351 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:16 smithi145 ceph-mon[24853]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 191 KiB/s rd, 584 B/s wr, 292 op/s; 6 B/s, 0 objects/s recovering 2024-03-24T11:36:16.351 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:16 smithi145 ceph-mon[24853]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-24T11:36:16.351 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:16 smithi145 ceph-mon[29394]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 191 KiB/s rd, 584 B/s wr, 292 op/s; 6 B/s, 0 objects/s recovering 2024-03-24T11:36:16.352 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:16 smithi145 ceph-mon[29394]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-24T11:36:16.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:16 smithi149 ceph-mon[25685]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 191 KiB/s rd, 584 B/s wr, 292 op/s; 6 B/s, 0 objects/s recovering 2024-03-24T11:36:16.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:16 smithi149 ceph-mon[25685]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-24T11:36:17.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:36:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:36:18.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[29394]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 5 B/s, 0 objects/s recovering 2024-03-24T11:36:18.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:18.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:18.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:18.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2501794633' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:36:18.226 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/732750250' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]: dispatch 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]: dispatch 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[24853]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 5 B/s, 0 objects/s recovering 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2501794633' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/732750250' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]: dispatch 2024-03-24T11:36:18.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:18 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]: dispatch 2024-03-24T11:36:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:18 smithi149 ceph-mon[25685]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 5 B/s, 0 objects/s recovering 2024-03-24T11:36:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:18 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:18 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:18 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:18 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2501794633' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:36:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:18 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/732750250' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]: dispatch 2024-03-24T11:36:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:18 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]: dispatch 2024-03-24T11:36:19.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:19 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]': finished 2024-03-24T11:36:19.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:19 smithi149 ceph-mon[25685]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:36:19.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:19 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2792460883' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2872389050"}]: dispatch 2024-03-24T11:36:19.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]': finished 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[24853]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2792460883' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2872389050"}]: dispatch 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/791882194"}]': finished 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[29394]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2792460883' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2872389050"}]: dispatch 2024-03-24T11:36:19.501 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[24853]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 345 B/s wr, 117 op/s; 5 B/s, 0 objects/s recovering 2024-03-24T11:36:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2792460883' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2872389050"}]': finished 2024-03-24T11:36:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[24853]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:36:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2026065595' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]: dispatch 2024-03-24T11:36:20.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]: dispatch 2024-03-24T11:36:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[29394]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 345 B/s wr, 117 op/s; 5 B/s, 0 objects/s recovering 2024-03-24T11:36:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2792460883' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2872389050"}]': finished 2024-03-24T11:36:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[29394]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:36:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2026065595' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]: dispatch 2024-03-24T11:36:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:20 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]: dispatch 2024-03-24T11:36:20.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:20 smithi149 ceph-mon[25685]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 345 B/s wr, 117 op/s; 5 B/s, 0 objects/s recovering 2024-03-24T11:36:20.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:20 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2792460883' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2872389050"}]': finished 2024-03-24T11:36:20.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:20 smithi149 ceph-mon[25685]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:36:20.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:20 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2026065595' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]: dispatch 2024-03-24T11:36:20.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:20 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]: dispatch 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]': finished 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/178875383' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2182110881"}]: dispatch 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-24T11:36:21.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-24T11:36:21.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi145"}]: dispatch 2024-03-24T11:36:21.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:21.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]': finished 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/178875383' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2182110881"}]: dispatch 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-24T11:36:21.515 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi145"}]: dispatch 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/1914003978"}]': finished 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/178875383' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2182110881"}]: dispatch 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.516 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.517 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-24T11:36:21.517 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-24T11:36:21.517 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.517 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi145"}]: dispatch 2024-03-24T11:36:21.517 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:21.517 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:21.517 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T11:36:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-24T11:36:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: Adding iSCSI gateway http://:@172.21.15.145:5000 to Dashboard 2024-03-24T11:36:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-24T11:36:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi145"}]: dispatch 2024-03-24T11:36:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/178875383' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2182110881"}]': finished 2024-03-24T11:36:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:36:22.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:22 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/432955603' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/1914003978"}]: dispatch 2024-03-24T11:36:22.569 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T11:36:22.569 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: Adding iSCSI gateway http://:@172.21.15.145:5000 to Dashboard 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi145"}]: dispatch 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/178875383' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2182110881"}]': finished 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/432955603' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/1914003978"}]: dispatch 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: Adding iSCSI gateway http://:@172.21.15.145:5000 to Dashboard 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-24T11:36:22.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi145"}]: dispatch 2024-03-24T11:36:22.571 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/178875383' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2182110881"}]': finished 2024-03-24T11:36:22.571 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:36:22.571 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:22 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/432955603' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/1914003978"}]: dispatch 2024-03-24T11:36:23.808 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:36:23.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:36:23.808 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:36:23.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/432955603' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/1914003978"}]': finished 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[29394]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2346958423' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]: dispatch 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]: dispatch 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[29394]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/432955603' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/1914003978"}]': finished 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[24853]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:36:23.809 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2346958423' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]: dispatch 2024-03-24T11:36:23.810 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]: dispatch 2024-03-24T11:36:23.810 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:23 smithi145 ceph-mon[24853]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:36:23.814 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:23 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/432955603' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/1914003978"}]': finished 2024-03-24T11:36:23.814 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:23 smithi149 ceph-mon[25685]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:36:23.814 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:23 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2346958423' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]: dispatch 2024-03-24T11:36:23.814 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:23 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]: dispatch 2024-03-24T11:36:23.815 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:23 smithi149 ceph-mon[25685]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-24T11:36:24.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:24] "GET /metrics HTTP/1.1" 200 214938 "" "Prometheus/2.33.4" 2024-03-24T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]': finished 2024-03-24T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1987526907' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]: dispatch 2024-03-24T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]: dispatch 2024-03-24T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: Checking dashboard <-> RGW credentials 2024-03-24T11:36:25.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:24 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]': finished 2024-03-24T11:36:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:36:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1987526907' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]: dispatch 2024-03-24T11:36:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]: dispatch 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: Checking dashboard <-> RGW credentials 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3876704990"}]': finished 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1987526907' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]: dispatch 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]: dispatch 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: Checking dashboard <-> RGW credentials 2024-03-24T11:36:25.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:24 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:36:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:25 smithi149 ceph-mon[25685]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8 op/s 2024-03-24T11:36:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:25 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]': finished 2024-03-24T11:36:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:25 smithi149 ceph-mon[25685]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:36:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:25 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/656999934' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3535593282"}]: dispatch 2024-03-24T11:36:26.265 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[29394]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8 op/s 2024-03-24T11:36:26.265 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]': finished 2024-03-24T11:36:26.265 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[29394]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:36:26.265 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/656999934' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3535593282"}]: dispatch 2024-03-24T11:36:26.265 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[24853]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8 op/s 2024-03-24T11:36:26.265 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2398354259"}]': finished 2024-03-24T11:36:26.265 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[24853]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:36:26.266 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:25 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/656999934' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3535593282"}]: dispatch 2024-03-24T11:36:27.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:26 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/656999934' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3535593282"}]': finished 2024-03-24T11:36:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:26 smithi145 ceph-mon[24853]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:36:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:26 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3096539517' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/877499941"}]: dispatch 2024-03-24T11:36:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:26 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/656999934' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3535593282"}]': finished 2024-03-24T11:36:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:26 smithi145 ceph-mon[29394]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:36:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:26 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3096539517' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/877499941"}]: dispatch 2024-03-24T11:36:27.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:26 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/656999934' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3535593282"}]': finished 2024-03-24T11:36:27.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:26 smithi149 ceph-mon[25685]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:36:27.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:26 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3096539517' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/877499941"}]: dispatch 2024-03-24T11:36:28.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:36:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:36:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:27 smithi149 ceph-mon[25685]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 8 op/s 2024-03-24T11:36:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:27 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3096539517' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/877499941"}]': finished 2024-03-24T11:36:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:27 smithi149 ceph-mon[25685]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:36:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:27 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1705877408' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/877499941"}]: dispatch 2024-03-24T11:36:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:27 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:28.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[29394]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 8 op/s 2024-03-24T11:36:28.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3096539517' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/877499941"}]': finished 2024-03-24T11:36:28.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[29394]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:36:28.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1705877408' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/877499941"}]: dispatch 2024-03-24T11:36:28.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[24853]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 8 op/s 2024-03-24T11:36:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3096539517' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/877499941"}]': finished 2024-03-24T11:36:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[24853]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:36:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1705877408' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/877499941"}]: dispatch 2024-03-24T11:36:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:27 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:28 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1705877408' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/877499941"}]': finished 2024-03-24T11:36:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:28 smithi149 ceph-mon[25685]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:36:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:28 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4278862648' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3212195359"}]: dispatch 2024-03-24T11:36:29.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:28 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1705877408' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/877499941"}]': finished 2024-03-24T11:36:29.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:28 smithi145 ceph-mon[29394]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:36:29.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:28 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4278862648' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3212195359"}]: dispatch 2024-03-24T11:36:29.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:28 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1705877408' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/877499941"}]': finished 2024-03-24T11:36:29.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:28 smithi145 ceph-mon[24853]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:36:29.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:28 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4278862648' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3212195359"}]: dispatch 2024-03-24T11:36:30.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:29 smithi149 ceph-mon[25685]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:30.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:29 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4278862648' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3212195359"}]': finished 2024-03-24T11:36:30.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:29 smithi149 ceph-mon[25685]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:36:30.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:29 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1012119828' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3212195359"}]: dispatch 2024-03-24T11:36:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[24853]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4278862648' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3212195359"}]': finished 2024-03-24T11:36:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[24853]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:36:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1012119828' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3212195359"}]: dispatch 2024-03-24T11:36:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[29394]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:36:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4278862648' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3212195359"}]': finished 2024-03-24T11:36:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[29394]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:36:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:29 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1012119828' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3212195359"}]: dispatch 2024-03-24T11:36:31.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:30 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1012119828' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3212195359"}]': finished 2024-03-24T11:36:31.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:30 smithi149 ceph-mon[25685]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:36:31.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:30 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2183231252' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]: dispatch 2024-03-24T11:36:31.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:30 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]: dispatch 2024-03-24T11:36:31.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1012119828' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3212195359"}]': finished 2024-03-24T11:36:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[29394]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:36:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2183231252' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]: dispatch 2024-03-24T11:36:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]: dispatch 2024-03-24T11:36:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1012119828' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3212195359"}]': finished 2024-03-24T11:36:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[24853]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:36:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2183231252' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]: dispatch 2024-03-24T11:36:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:30 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]: dispatch 2024-03-24T11:36:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:31 smithi149 ceph-mon[25685]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:31 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]': finished 2024-03-24T11:36:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:31 smithi149 ceph-mon[25685]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:36:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:31 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3860308030' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3234105728"}]: dispatch 2024-03-24T11:36:32.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[29394]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:32.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]': finished 2024-03-24T11:36:32.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[29394]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:36:32.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3860308030' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3234105728"}]: dispatch 2024-03-24T11:36:32.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[24853]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:32.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1228964895"}]': finished 2024-03-24T11:36:32.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[24853]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:36:32.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:31 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3860308030' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3234105728"}]: dispatch 2024-03-24T11:36:33.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:32 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3860308030' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3234105728"}]': finished 2024-03-24T11:36:33.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:32 smithi149 ceph-mon[25685]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:36:33.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:32 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3685665308' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]: dispatch 2024-03-24T11:36:33.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:32 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]: dispatch 2024-03-24T11:36:33.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3860308030' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3234105728"}]': finished 2024-03-24T11:36:33.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[29394]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:36:33.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3685665308' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]: dispatch 2024-03-24T11:36:33.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]: dispatch 2024-03-24T11:36:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3860308030' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3234105728"}]': finished 2024-03-24T11:36:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[24853]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:36:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3685665308' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]: dispatch 2024-03-24T11:36:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:32 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]: dispatch 2024-03-24T11:36:33.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:36:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:36:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:33 smithi149 ceph-mon[25685]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:33 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]': finished 2024-03-24T11:36:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:33 smithi149 ceph-mon[25685]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:36:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:33 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3324050644' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/195323836"}]: dispatch 2024-03-24T11:36:34.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:36:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:34] "GET /metrics HTTP/1.1" 200 214938 "" "Prometheus/2.33.4" 2024-03-24T11:36:34.388 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[29394]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:34.388 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]': finished 2024-03-24T11:36:34.389 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[29394]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:36:34.389 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3324050644' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/195323836"}]: dispatch 2024-03-24T11:36:34.389 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[24853]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:34.389 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2888711157"}]': finished 2024-03-24T11:36:34.389 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[24853]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:36:34.389 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:33 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3324050644' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/195323836"}]: dispatch 2024-03-24T11:36:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:34 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3324050644' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/195323836"}]': finished 2024-03-24T11:36:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:34 smithi149 ceph-mon[25685]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:36:35.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:34 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3324050644' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/195323836"}]': finished 2024-03-24T11:36:35.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:34 smithi145 ceph-mon[29394]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:36:35.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:34 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3324050644' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/195323836"}]': finished 2024-03-24T11:36:35.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:34 smithi145 ceph-mon[24853]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:36:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:35 smithi149 ceph-mon[25685]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:36.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:35 smithi145 ceph-mon[24853]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:36.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:35 smithi145 ceph-mon[29394]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:37 smithi149 ceph-mon[25685]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-24T11:36:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:37 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:38.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:36:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:36:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:37 smithi145 ceph-mon[29394]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-24T11:36:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:37 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:37 smithi145 ceph-mon[24853]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-24T11:36:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:37 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:39 smithi149 ceph-mon[25685]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-24T11:36:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:39 smithi145 ceph-mon[24853]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-24T11:36:40.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:39 smithi145 ceph-mon[29394]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-24T11:36:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:41 smithi149 ceph-mon[25685]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:42.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:41 smithi145 ceph-mon[24853]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:41 smithi145 ceph-mon[29394]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:43 smithi149 ceph-mon[25685]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:44.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:43 smithi145 ceph-mon[24853]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:44.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:43 smithi145 ceph-mon[29394]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:44.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:36:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:44] "GET /metrics HTTP/1.1" 200 214949 "" "Prometheus/2.33.4" 2024-03-24T11:36:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:45 smithi149 ceph-mon[25685]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:36:46.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:45 smithi145 ceph-mon[29394]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:36:46.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:45 smithi145 ceph-mon[24853]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:36:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:47 smithi149 ceph-mon[25685]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:47 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:48.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:36:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:36:48.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:47 smithi145 ceph-mon[24853]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:48.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:47 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:47 smithi145 ceph-mon[29394]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:47 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:50.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:49 smithi149 ceph-mon[25685]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:50.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:49 smithi145 ceph-mon[24853]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:50.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:49 smithi145 ceph-mon[29394]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:51 smithi149 ceph-mon[25685]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:52.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:51 smithi145 ceph-mon[29394]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:52.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:51 smithi145 ceph-mon[24853]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:54.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:53 smithi149 ceph-mon[25685]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:54.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:36:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:54] "GET /metrics HTTP/1.1" 200 214896 "" "Prometheus/2.33.4" 2024-03-24T11:36:54.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:53 smithi145 ceph-mon[29394]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:54.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:53 smithi145 ceph-mon[24853]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:55 smithi149 ceph-mon[25685]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:56.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:55 smithi145 ceph-mon[29394]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:56.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:55 smithi145 ceph-mon[24853]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:57 smithi149 ceph-mon[25685]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:57 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:58.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:36:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:36:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:36:58.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:57 smithi145 ceph-mon[29394]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:58.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:57 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:57 smithi145 ceph-mon[24853]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:57 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:00.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:36:59 smithi149 ceph-mon[25685]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:00.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:36:59 smithi145 ceph-mon[29394]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:00.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:36:59 smithi145 ceph-mon[24853]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:01 smithi149 ceph-mon[25685]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:02.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:01 smithi145 ceph-mon[24853]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:02.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:01 smithi145 ceph-mon[29394]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:03 smithi149 ceph-mon[25685]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:04.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:03 smithi145 ceph-mon[24853]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:03 smithi145 ceph-mon[29394]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:04.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:37:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:04] "GET /metrics HTTP/1.1" 200 214896 "" "Prometheus/2.33.4" 2024-03-24T11:37:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:05 smithi149 ceph-mon[25685]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:06.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:05 smithi145 ceph-mon[29394]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:06.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:05 smithi145 ceph-mon[24853]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:07 smithi149 ceph-mon[25685]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:07 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:08.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:37:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:37:08.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:07 smithi145 ceph-mon[29394]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:08.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:07 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:08.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:07 smithi145 ceph-mon[24853]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:08.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:07 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:37:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:37:09.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:37:09.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:37:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:37:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:37:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:09 smithi149 ceph-mon[25685]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:10.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:09 smithi145 ceph-mon[29394]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:10.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:09 smithi145 ceph-mon[24853]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:12.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:11 smithi149 ceph-mon[25685]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:12.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:11 smithi145 ceph-mon[29394]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:12.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:11 smithi145 ceph-mon[24853]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:13 smithi149 ceph-mon[25685]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:14.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:13 smithi145 ceph-mon[24853]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:14.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:37:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:14] "GET /metrics HTTP/1.1" 200 214849 "" "Prometheus/2.33.4" 2024-03-24T11:37:14.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:13 smithi145 ceph-mon[29394]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:15 smithi149 ceph-mon[25685]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:16.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:15 smithi145 ceph-mon[24853]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:16.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:15 smithi145 ceph-mon[29394]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:17 smithi149 ceph-mon[25685]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:17 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:18.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:37:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:37:18.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:17 smithi145 ceph-mon[29394]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:18.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:17 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:17 smithi145 ceph-mon[24853]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:17 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:19 smithi149 ceph-mon[25685]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:20.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:19 smithi145 ceph-mon[29394]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:19 smithi145 ceph-mon[24853]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:22.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:21 smithi149 ceph-mon[25685]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:22.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:21 smithi145 ceph-mon[29394]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:22.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:21 smithi145 ceph-mon[24853]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:23 smithi149 ceph-mon[25685]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:24.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:23 smithi145 ceph-mon[29394]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:24.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:23 smithi145 ceph-mon[24853]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:24.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:37:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:24] "GET /metrics HTTP/1.1" 200 214854 "" "Prometheus/2.33.4" 2024-03-24T11:37:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:24 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:24 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:25.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:24 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:25.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:24 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:24 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:24 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:25 smithi149 ceph-mon[25685]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:25 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:37:26.263 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:25 smithi145 ceph-mon[24853]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:26.263 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:25 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:37:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:25 smithi145 ceph-mon[29394]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:25 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:37:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:27 smithi145 ceph-mon[24853]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:27 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:27 smithi145 ceph-mon[29394]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:27 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:28.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:37:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:37:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:27 smithi149 ceph-mon[25685]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:27 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:30.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:29 smithi149 ceph-mon[25685]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:29 smithi145 ceph-mon[24853]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:29 smithi145 ceph-mon[29394]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:31 smithi149 ceph-mon[25685]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:32.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:31 smithi145 ceph-mon[24853]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:32.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:31 smithi145 ceph-mon[29394]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:33 smithi149 ceph-mon[25685]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:34.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:33 smithi145 ceph-mon[29394]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:34.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:33 smithi145 ceph-mon[24853]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:34.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:37:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:34] "GET /metrics HTTP/1.1" 200 214854 "" "Prometheus/2.33.4" 2024-03-24T11:37:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:35 smithi149 ceph-mon[25685]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:36.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:35 smithi145 ceph-mon[29394]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:36.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:35 smithi145 ceph-mon[24853]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:37 smithi149 ceph-mon[25685]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:37 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:38.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:37:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:37:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:37 smithi145 ceph-mon[29394]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:37 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:37 smithi145 ceph-mon[24853]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:38.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:37 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:39 smithi149 ceph-mon[25685]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:40.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:39 smithi145 ceph-mon[29394]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:40.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:39 smithi145 ceph-mon[24853]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:41 smithi149 ceph-mon[25685]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:41 smithi145 ceph-mon[29394]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:41 smithi145 ceph-mon[24853]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:43 smithi149 ceph-mon[25685]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:44.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:43 smithi145 ceph-mon[29394]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:44.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:43 smithi145 ceph-mon[24853]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:44.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:37:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:44] "GET /metrics HTTP/1.1" 200 214856 "" "Prometheus/2.33.4" 2024-03-24T11:37:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:45 smithi149 ceph-mon[25685]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:46.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:45 smithi145 ceph-mon[24853]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:46.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:45 smithi145 ceph-mon[29394]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:47 smithi149 ceph-mon[25685]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:47 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:48.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:37:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:37:48.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:47 smithi145 ceph-mon[24853]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:48.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:47 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:47 smithi145 ceph-mon[29394]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:47 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:50.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:49 smithi149 ceph-mon[25685]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:50.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:49 smithi145 ceph-mon[29394]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:50.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:49 smithi145 ceph-mon[24853]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:52.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:51 smithi149 ceph-mon[25685]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:52.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:51 smithi145 ceph-mon[29394]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:52.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:51 smithi145 ceph-mon[24853]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:54.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:53 smithi149 ceph-mon[25685]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:54.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:53 smithi145 ceph-mon[29394]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:54.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:37:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:54] "GET /metrics HTTP/1.1" 200 214861 "" "Prometheus/2.33.4" 2024-03-24T11:37:54.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:53 smithi145 ceph-mon[24853]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:55 smithi149 ceph-mon[25685]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:56.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:55 smithi145 ceph-mon[29394]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:56.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:55 smithi145 ceph-mon[24853]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:37:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:57 smithi149 ceph-mon[25685]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:57 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:58.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:37:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:37:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:37:58.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:57 smithi145 ceph-mon[29394]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:58.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:57 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:57 smithi145 ceph-mon[24853]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:37:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:57 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:00.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:37:59 smithi149 ceph-mon[25685]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:00.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:37:59 smithi145 ceph-mon[29394]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:37:59 smithi145 ceph-mon[24853]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:02.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:01 smithi149 ceph-mon[25685]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:02.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:01 smithi145 ceph-mon[24853]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:02.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:01 smithi145 ceph-mon[29394]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:03 smithi149 ceph-mon[25685]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:04.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:03 smithi145 ceph-mon[24853]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:04.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:03 smithi145 ceph-mon[29394]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:04.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:38:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:04] "GET /metrics HTTP/1.1" 200 214861 "" "Prometheus/2.33.4" 2024-03-24T11:38:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:05 smithi149 ceph-mon[25685]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:06.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:05 smithi145 ceph-mon[24853]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:06.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:05 smithi145 ceph-mon[29394]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:08.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:38:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:38:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:07 smithi149 ceph-mon[25685]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:08.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:07 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:08.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:07 smithi145 ceph-mon[29394]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:08.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:07 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:08.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:07 smithi145 ceph-mon[24853]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:08.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:07 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:38:09.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:38:09.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:38:09.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:38:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:38:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:38:10.086 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:09 smithi145 ceph-mon[24853]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:10.086 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:09 smithi145 ceph-mon[29394]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:09 smithi149 ceph-mon[25685]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:10.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-24T11:38:12.298 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:11 smithi145 ceph-mon[29394]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:12.299 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:11 smithi145 ceph-mon[24853]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:11 smithi149 ceph-mon[25685]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:12.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-24T11:38:14.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:13 smithi145 ceph-mon[24853]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:14.168 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:13 smithi145 ceph-mon[29394]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:14.169 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:38:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:14] "GET /metrics HTTP/1.1" 200 214876 "" "Prometheus/2.33.4" 2024-03-24T11:38:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:13 smithi149 ceph-mon[25685]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:15.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-24T11:38:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:15 smithi149 ceph-mon[25685]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:16.227 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:15 smithi145 ceph-mon[29394]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:16.227 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:15 smithi145 ceph-mon[24853]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:17.725 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-24T11:38:18.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:17 smithi145 ceph-mon[24853]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:18.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:17 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:18.029 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:17 smithi145 ceph-mon[29394]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:18.029 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:17 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:17 smithi149 ceph-mon[25685]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:17 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:18.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:38:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:38:19.616 INFO:teuthology.orchestra.run.smithi145.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:38:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:19 smithi149 ceph-mon[25685]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:19 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[29394]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[24853]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:20.231 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:19 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:20.255 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:38:20.265 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-24T11:38:20.265 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-24T11:38:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:20 smithi145 ceph-mon[29394]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:20 smithi145 ceph-mon[29394]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:38:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:20 smithi145 ceph-mon[29394]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:38:21.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:20 smithi145 ceph-mon[24853]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:21.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:20 smithi145 ceph-mon[24853]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:38:21.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:20 smithi145 ceph-mon[24853]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:38:21.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:20 smithi149 ceph-mon[25685]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:21.478 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:20 smithi149 ceph-mon[25685]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:38:21.478 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:20 smithi149 ceph-mon[25685]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:38:22.210 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:21 smithi145 ceph-mon[29394]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:22.210 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:21 smithi145 ceph-mon[24853]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:22.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:21 smithi149 ceph-mon[25685]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:22.612 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:38:23.482 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (2m) 119s ago 3m 15.5M - ba2b418f427c b0ee65c327d9 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (2m) 119s ago 2m 46.7M - 8.3.5 dad864ee21e9 1ea96d14b839 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (2m) 119s ago 2m 76.8M - 3.5 e1d6a67b021e 149865107de4 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443 running (7m) 119s ago 7m 411M - 17.2.0 e1d6a67b021e 252e0f23bfd2 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:9283 running (9m) 119s ago 9m 456M - 17.2.0 e1d6a67b021e 642048ee9b24 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (9m) 119s ago 9m 48.0M 2048M 17.2.0 e1d6a67b021e ca31a24577e6 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (7m) 119s ago 7m 40.4M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (8m) 119s ago 8m 38.7M 2048M 17.2.0 e1d6a67b021e 0b79dbdf9a4c 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (3m) 119s ago 3m 19.1M - 1dbe0e931976 7399120d8cc1 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (3m) 119s ago 3m 16.5M - 1dbe0e931976 edb4947ca62a 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (7m) 119s ago 7m 48.3M 4016M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (6m) 119s ago 6m 50.1M 4016M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:38:23.483 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (6m) 119s ago 6m 45.9M 4016M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (6m) 119s ago 6m 46.6M 4016M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (5m) 119s ago 5m 50.1M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (5m) 119s ago 5m 46.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (5m) 119s ago 5m 44.1M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (4m) 119s ago 4m 48.1M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (2m) 119s ago 3m 44.6M - 514e6a882f6e b35d6bc7db17 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (2m) 119s ago 2m 67.8M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (2m) 119s ago 2m 71.6M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (2m) 119s ago 2m 70.0M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:38:23.484 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (2m) 119s ago 2m 70.4M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:38:23.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": {}, 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:38:23.965 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:38:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:23 smithi149 ceph-mon[25685]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:23 smithi149 ceph-mon[25685]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:23 smithi149 ceph-mon[25685]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:24.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:24] "GET /metrics HTTP/1.1" 200 214869 "" "Prometheus/2.33.4" 2024-03-24T11:38:24.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:23 smithi145 ceph-mon[29394]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:24.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:23 smithi145 ceph-mon[29394]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:24.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:23 smithi145 ceph-mon[29394]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:24.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:23 smithi145 ceph-mon[24853]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:24.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:23 smithi145 ceph-mon[24853]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:24.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:23 smithi145 ceph-mon[24853]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:24.365 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:38:24.366 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:38:24.366 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:38:24.366 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [], 2024-03-24T11:38:24.366 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "", 2024-03-24T11:38:24.366 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 image" 2024-03-24T11:38:24.366 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:38:24.840 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:38:25.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[29394]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:25.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3458729896' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:38:25.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[29394]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2140532652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:38:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[24853]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3458729896' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:38:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[24853]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:24 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2140532652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:38:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:24 smithi149 ceph-mon[25685]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:24 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3458729896' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:38:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:24 smithi149 ceph-mon[25685]: from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:24 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2140532652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:38:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:25 smithi149 ceph-mon[25685]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:26.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:25 smithi145 ceph-mon[24853]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:26.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:25 smithi145 ceph-mon[29394]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:27 smithi149 ceph-mon[25685]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:27 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:28.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:38:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:38:28.296 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:27 smithi145 ceph-mon[24853]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:28.296 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:27 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:28.296 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:27 smithi145 ceph-mon[29394]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:28.296 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:27 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:30.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:29 smithi149 ceph-mon[25685]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:29 smithi145 ceph-mon[24853]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:30.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:29 smithi145 ceph-mon[29394]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:32.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:31 smithi149 ceph-mon[25685]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:32.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:31 smithi145 ceph-mon[24853]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:32.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:31 smithi145 ceph-mon[29394]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:33 smithi149 ceph-mon[25685]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:34.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:38:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:34] "GET /metrics HTTP/1.1" 200 214869 "" "Prometheus/2.33.4" 2024-03-24T11:38:34.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:33 smithi145 ceph-mon[29394]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:33 smithi145 ceph-mon[24853]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:36.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:35 smithi149 ceph-mon[25685]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:36.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:35 smithi145 ceph-mon[24853]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:36.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:35 smithi145 ceph-mon[29394]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:37.976 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:37 smithi145 ceph-mon[24853]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:37.976 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:37 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:37 smithi149 ceph-mon[25685]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:38.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:37 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:38.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:38:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:38:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:37 smithi145 ceph-mon[29394]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:38.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:37 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:40.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:39 smithi145 ceph-mon[24853]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:40.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:39 smithi145 ceph-mon[29394]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:40.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:39 smithi149 ceph-mon[25685]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:41 smithi145 ceph-mon[29394]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:42.382 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:41 smithi145 ceph-mon[24853]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:42.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:41 smithi149 ceph-mon[25685]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:43 smithi149 ceph-mon[25685]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:44.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:43 smithi145 ceph-mon[29394]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:44.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:43 smithi145 ceph-mon[24853]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:44.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:38:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:44] "GET /metrics HTTP/1.1" 200 214831 "" "Prometheus/2.33.4" 2024-03-24T11:38:46.090 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:45 smithi145 ceph-mon[29394]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:46.090 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:45 smithi145 ceph-mon[24853]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:45 smithi149 ceph-mon[25685]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:47.369 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:47.369 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[24853]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T11:38:47.369 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[24853]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T11:38:47.369 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:47.369 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[24853]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[24853]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on smithi149 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[29394]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[29394]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[29394]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-24T11:38:47.370 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:47 smithi145 ceph-mon[29394]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on smithi149 2024-03-24T11:38:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-mon[25685]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T11:38:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-mon[25685]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T11:38:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:38:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-mon[25685]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-24T11:38:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-mon[25685]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on smithi149 2024-03-24T11:38:48.133 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:38:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:38:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:48 smithi149 ceph-mon[25685]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:48 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:48.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:48 smithi145 ceph-mon[29394]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:48.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:48 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:48.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:48 smithi145 ceph-mon[24853]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:48.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:48 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:50.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:50 smithi149 ceph-mon[25685]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:50.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:50 smithi145 ceph-mon[29394]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:50.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:50 smithi145 ceph-mon[24853]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:52.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:52 smithi149 ceph-mon[25685]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:52.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:52 smithi145 ceph-mon[29394]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:52.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:52 smithi145 ceph-mon[24853]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:53.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:38:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:38:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:38:53.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:38:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:38:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:38:54.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:53 smithi149 ceph-mon[25685]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:54.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:53 smithi145 ceph-mon[29394]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:54.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:53 smithi145 ceph-mon[24853]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:54.363 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:38:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:54] "GET /metrics HTTP/1.1" 200 214814 "" "Prometheus/2.33.4" 2024-03-24T11:38:55.177 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:38:55.772 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:38:55.772 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (3m) 2m ago 3m 15.5M - ba2b418f427c b0ee65c327d9 2024-03-24T11:38:55.772 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (3m) 2m ago 3m 46.7M - 8.3.5 dad864ee21e9 1ea96d14b839 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (2m) 2m ago 2m 76.8M - 3.5 e1d6a67b021e 149865107de4 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 252e0f23bfd2 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 642048ee9b24 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (9m) 2m ago 9m 48.0M 2048M 17.2.0 e1d6a67b021e ca31a24577e6 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (8m) 2m ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (8m) 2m ago 8m 38.7M 2048M 17.2.0 e1d6a67b021e 0b79dbdf9a4c 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (4m) 2m ago 4m 19.1M - 1dbe0e931976 7399120d8cc1 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (4m) 2m ago 4m 16.5M - 1dbe0e931976 edb4947ca62a 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (7m) 2m ago 7m 48.3M 4016M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (7m) 2m ago 7m 50.1M 4016M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (7m) 2m ago 7m 45.9M 4016M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (6m) 2m ago 6m 46.6M 4016M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:38:55.773 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (6m) 2m ago 6m 50.1M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (5m) 2m ago 5m 46.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (5m) 2m ago 5m 44.1M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (5m) 2m ago 5m 48.1M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (3m) 2m ago 4m 44.6M - 514e6a882f6e b35d6bc7db17 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (2m) 2m ago 2m 67.8M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (2m) 2m ago 2m 71.6M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (2m) 2m ago 2m 70.0M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:38:55.774 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (2m) 2m ago 2m 70.4M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:38:56.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:55 smithi145 ceph-mon[29394]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:56.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:55 smithi145 ceph-mon[29394]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:56.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:55 smithi145 ceph-mon[24853]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:56.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:55 smithi145 ceph-mon[24853]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:56.121 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:38:56.121 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": {}, 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:38:56.122 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-24T11:38:56.123 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:38:56.123 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:38:56.427 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:38:56.427 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:38:56.427 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:38:56.428 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [], 2024-03-24T11:38:56.428 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "0/23 daemons upgraded", 2024-03-24T11:38:56.428 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 image on host smithi149" 2024-03-24T11:38:56.428 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:38:56.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:55 smithi149 ceph-mon[25685]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:38:56.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:55 smithi149 ceph-mon[25685]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:56.843 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:38:57.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[29394]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[29394]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2650526459' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:38:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[29394]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/401463196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:38:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[24853]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[24853]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2650526459' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:38:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[24853]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:56 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/401463196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:38:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:56 smithi149 ceph-mon[25685]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:56 smithi149 ceph-mon[25685]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:56 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2650526459' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:38:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:56 smithi149 ceph-mon[25685]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:38:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:56 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/401463196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:38:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:57 smithi149 ceph-mon[25685]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:57 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:58.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:38:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:38:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:38:58.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:57 smithi145 ceph-mon[29394]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:58.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:57 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:57 smithi145 ceph-mon[24853]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:38:58.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:57 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:00.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:38:59 smithi145 ceph-mon[29394]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:38:59 smithi145 ceph-mon[24853]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:00.474 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:38:59 smithi149 ceph-mon[25685]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:02.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:01 smithi145 ceph-mon[29394]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:02.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:01 smithi145 ceph-mon[24853]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:02.474 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:01 smithi149 ceph-mon[25685]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:03.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:39:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:39:03.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:03.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:39:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:03 smithi149 ceph-mon[25685]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:04.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:03 smithi145 ceph-mon[24853]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:03 smithi145 ceph-mon[29394]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:04.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:04] "GET /metrics HTTP/1.1" 200 214814 "" "Prometheus/2.33.4" 2024-03-24T11:39:06.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:05 smithi145 ceph-mon[29394]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:06.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:05 smithi145 ceph-mon[24853]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:05 smithi149 ceph-mon[25685]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:07 smithi149 ceph-mon[25685]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:07 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:08.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:39:08.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:07 smithi145 ceph-mon[24853]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:08.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:07 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:07 smithi145 ceph-mon[29394]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:07 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:08 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:09.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:08 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:09.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:09.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:08 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:10.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:09 smithi145 ceph-mon[29394]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:10.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:09 smithi145 ceph-mon[24853]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:10.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:09 smithi149 ceph-mon[25685]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:12.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:11 smithi145 ceph-mon[24853]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:12.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:11 smithi145 ceph-mon[29394]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:11 smithi149 ceph-mon[25685]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:13.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:13 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:39:13.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:13.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:13 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:13.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:13 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:39:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:13 smithi149 ceph-mon[25685]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:14.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:13 smithi145 ceph-mon[29394]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:14.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:13 smithi145 ceph-mon[24853]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:14.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:14] "GET /metrics HTTP/1.1" 200 214811 "" "Prometheus/2.33.4" 2024-03-24T11:39:16.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:15 smithi145 ceph-mon[29394]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:16.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:15 smithi145 ceph-mon[24853]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:16.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:15 smithi149 ceph-mon[25685]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:17 smithi149 ceph-mon[25685]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:17 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:18.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[26870]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:39:18.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:17 smithi145 ceph-mon[29394]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:18.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:17 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:17 smithi145 ceph-mon[24853]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:17 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:20.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:19 smithi149 ceph-mon[25685]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:20.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:19 smithi145 ceph-mon[24853]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:19 smithi145 ceph-mon[29394]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:21.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[29394]: Upgrade: Updating mgr.x 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[29394]: Deploying daemon mgr.x on smithi149 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[24853]: Upgrade: Updating mgr.x 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:39:21.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:21.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:21 smithi145 ceph-mon[24853]: Deploying daemon mgr.x on smithi149 2024-03-24T11:39:21.682 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:21 smithi149 ceph-mon[25685]: Upgrade: Updating mgr.x 2024-03-24T11:39:21.683 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:21.683 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:39:21.683 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:39:21.683 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:21 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:21.683 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:21 smithi149 ceph-mon[25685]: Deploying daemon mgr.x on smithi149 2024-03-24T11:39:21.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:21 smithi149 systemd[1]: Stopping Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:39:21.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:21 smithi149 bash[46784]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-24T11:39:22.329 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 podman[46797]: 2024-03-24 11:39:22.040431669 +0000 UTC m=+0.117606485 container died 252e0f23bfd23cb23cd041c6278c76920db669ee5641777d66ab808cbfc30f00 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc.) 2024-03-24T11:39:22.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:22 smithi145 ceph-mon[29394]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:22.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:22 smithi145 ceph-mon[24853]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:22 smithi149 ceph-mon[25685]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:22.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 podman[46797]: 2024-03-24 11:39:22.511381093 +0000 UTC m=+0.588555905 container cleanup 252e0f23bfd23cb23cd041c6278c76920db669ee5641777d66ab808cbfc30f00 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, maintainer=Guillaume Abrioux , name=centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870) 2024-03-24T11:39:22.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 bash[46797]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x 2024-03-24T11:39:22.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 bash[46824]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-24T11:39:23.201 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 podman[46811]: 2024-03-24 11:39:22.779726739 +0000 UTC m=+0.732494506 container remove 252e0f23bfd23cb23cd041c6278c76920db669ee5641777d66ab808cbfc30f00 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-type=git, version=8, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-24T11:39:23.201 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:39:23.201 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-24T11:39:23.201 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 systemd[1]: Stopped Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:39:23.201 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service: Consumed 16.475s CPU time. 2024-03-24T11:39:23.201 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:22 smithi149 systemd[1]: Starting Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:39:23.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:23 smithi149 podman[46930]: 2024-03-24 11:39:23.100525545 +0000 UTC m=+0.017771304 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:39:23.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:23 smithi149 podman[46930]: 2024-03-24 11:39:23.22751823 +0000 UTC m=+0.144763974 container create 25419047a080aab568683f6dac6fd20669e83a3b5d5c621ee22e68d24560b2a8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-03-24T11:39:23.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:23 smithi149 podman[46930]: 2024-03-24 11:39:23.474896765 +0000 UTC m=+0.392142520 container init 25419047a080aab568683f6dac6fd20669e83a3b5d5c621ee22e68d24560b2a8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T11:39:23.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:23 smithi149 podman[46930]: 2024-03-24 11:39:23.477753182 +0000 UTC m=+0.394998936 container start 25419047a080aab568683f6dac6fd20669e83a3b5d5c621ee22e68d24560b2a8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T11:39:23.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:23 smithi149 ceph-mgr[46949]: -- 172.21.15.149:0/2460512668 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56489c8a81a0 con 0x56489c87e800 2024-03-24T11:39:23.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:23 smithi149 bash[46930]: 25419047a080aab568683f6dac6fd20669e83a3b5d5c621ee22e68d24560b2a8 2024-03-24T11:39:23.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:23 smithi149 systemd[1]: Started Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:39:23.782 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:39:23.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:23.783 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:39:23.783 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:24.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:23 smithi149 ceph-mon[25685]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:24.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:23 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:24.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:23 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:24.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:23 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:24.070 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:24] "GET /metrics HTTP/1.1" 200 214809 "" "Prometheus/2.33.4" 2024-03-24T11:39:24.070 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[24853]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:24.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:24.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:24.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:24.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[29394]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:24.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:24.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:24.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:23 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:24.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:24.047+0000 7fef4b4e8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:39:24.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:24.159+0000 7fef4b4e8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:39:25.063 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:24.811+0000 7fef4b4e8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:39:25.063 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:24.909+0000 7fef4b4e8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:39:25.358 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:25.062+0000 7fef4b4e8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:39:25.358 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:25.165+0000 7fef4b4e8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:39:25.650 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:25.256+0000 7fef4b4e8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:39:25.650 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:25.535+0000 7fef4b4e8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:39:25.650 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:25.649+0000 7fef4b4e8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:25.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:25.738+0000 7fef4b4e8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:39:25.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:25.836+0000 7fef4b4e8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:39:26.254 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:25 smithi149 ceph-mon[25685]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:26.254 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:26 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:26.021+0000 7fef4b4e8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:39:26.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:25 smithi145 ceph-mon[29394]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:26.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:25 smithi145 ceph-mon[24853]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:27.171 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:39:27.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:27.022+0000 7fef4b4e8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:39:27.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:27.154+0000 7fef4b4e8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:39:27.792 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:27.502+0000 7fef4b4e8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:39:27.792 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:27.593+0000 7fef4b4e8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:39:27.793 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:27.791+0000 7fef4b4e8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:27.796 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:39:27.796 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (3m) 3m ago 4m 15.5M - ba2b418f427c b0ee65c327d9 2024-03-24T11:39:27.796 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (3m) 3m ago 3m 46.7M - 8.3.5 dad864ee21e9 1ea96d14b839 2024-03-24T11:39:27.796 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (3m) 3m ago 3m 76.8M - 3.5 e1d6a67b021e 149865107de4 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283 starting - - - - 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 642048ee9b24 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (10m) 3m ago 10m 48.0M 2048M 17.2.0 e1d6a67b021e ca31a24577e6 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (8m) 3m ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (9m) 3m ago 9m 38.7M 2048M 17.2.0 e1d6a67b021e 0b79dbdf9a4c 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (4m) 3m ago 4m 19.1M - 1dbe0e931976 7399120d8cc1 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (4m) 3m ago 4m 16.5M - 1dbe0e931976 edb4947ca62a 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (8m) 3m ago 8m 48.3M 4016M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (8m) 3m ago 8m 50.1M 4016M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:39:27.797 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (7m) 3m ago 7m 45.9M 4016M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:39:27.798 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (7m) 3m ago 7m 46.6M 4016M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:39:27.798 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (6m) 3m ago 6m 50.1M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:39:27.798 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (6m) 3m ago 6m 46.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:39:27.798 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (6m) 3m ago 6m 44.1M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:39:27.798 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (5m) 3m ago 5m 48.1M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:39:27.798 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (3m) 3m ago 5m 44.6M - 514e6a882f6e b35d6bc7db17 2024-03-24T11:39:27.798 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (3m) 3m ago 3m 67.8M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:39:27.799 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (3m) 3m ago 3m 71.6M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:39:27.799 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (3m) 3m ago 3m 70.0M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:39:27.799 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (3m) 3m ago 3m 70.4M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:39:28.043 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-mon[25685]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:28.043 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-mon[25685]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:28.043 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:27 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:28.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:27 smithi145 ceph-mon[29394]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:28.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:27 smithi145 ceph-mon[29394]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:28.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:27 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:27 smithi145 ceph-mon[24853]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:27 smithi145 ceph-mon[24853]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:27 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:28.142 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:39:28.142 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": {}, 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:39:28.143 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-24T11:39:28.144 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:39:28.144 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:39:28.436 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:39:28.436 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:39:28.436 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:39:28.436 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [], 2024-03-24T11:39:28.436 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "", 2024-03-24T11:39:28.436 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading mgr daemons" 2024-03-24T11:39:28.437 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:39:28.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:27.980+0000 7fef4b4e8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:39:28.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:28.268+0000 7fef4b4e8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:28.777 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:39:28.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:28.536+0000 7fef4b4e8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:28.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:28.772+0000 7fef4b4e8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:39:29.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[24853]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[24853]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4263882641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[24853]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1319079861' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[29394]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[29394]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4263882641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[29394]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:28 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1319079861' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:39:29.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-mon[25685]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-mon[25685]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4263882641' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:39:29.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-mon[25685]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:39:29.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:28 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1319079861' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:39:29.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:29 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:29.587+0000 7fef4b4e8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:39:30.294 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:29 smithi149 ceph-mon[25685]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:30.295 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:30.181+0000 7fef4b4e8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:39:30.295 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:30.293+0000 7fef4b4e8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:39:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:29 smithi145 ceph-mon[24853]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:30.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:29 smithi145 ceph-mon[29394]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:30.631 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:30.511+0000 7fef4b4e8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:39:30.631 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:30] ENGINE Bus STARTING 2024-03-24T11:39:30.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: CherryPy Checker: 2024-03-24T11:39:30.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: The Application mounted at '' has an empty config. 2024-03-24T11:39:30.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:39:30.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:30] ENGINE Serving on http://:::9283 2024-03-24T11:39:30.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:30] ENGINE Bus STARTED 2024-03-24T11:39:31.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[29394]: Standby manager daemon x restarted 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[29394]: Standby manager daemon x started 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:30 smithi145 ceph-mon[24853]: Standby manager daemon x restarted 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[24853]: Standby manager daemon x started 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:39:31.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:39:31.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:39:31.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:31 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:39:31.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:31 smithi149 ceph-mon[25685]: Standby manager daemon x restarted 2024-03-24T11:39:31.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:31 smithi149 ceph-mon[25685]: Standby manager daemon x started 2024-03-24T11:39:31.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:31 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:39:31.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:31 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:39:31.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:31 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:39:31.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:31 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:39:32.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:32 smithi145 ceph-mon[29394]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:32.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:32 smithi145 ceph-mon[29394]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-24T11:39:32.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:32 smithi145 ceph-mon[24853]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:32.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:32 smithi145 ceph-mon[24853]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-24T11:39:32.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:32 smithi149 ceph-mon[25685]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:32.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:32 smithi149 ceph-mon[25685]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-24T11:39:33.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:39:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:33.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:33.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:39:34.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:33 smithi149 ceph-mon[25685]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:34.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:33 smithi145 ceph-mon[29394]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:34.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:33 smithi145 ceph-mon[24853]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:34.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:34] "GET /metrics HTTP/1.1" 200 214809 "" "Prometheus/2.33.4" 2024-03-24T11:39:36.097 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:35 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:36.097 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:35 smithi149 ceph-mon[25685]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:36.097 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:35 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:35 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:35 smithi145 ceph-mon[24853]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:35 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:35 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:35 smithi145 ceph-mon[29394]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:39:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:35 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:38.222 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:37 smithi149 ceph-mon[25685]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:38.222 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:37 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:38.222 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:39:38.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:37 smithi145 ceph-mon[24853]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:38.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:37 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:37 smithi145 ceph-mon[29394]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:39:38.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:37 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: Failing over to other MGR 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:39:40.143 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[29394]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: Failing over to other MGR 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mon[24853]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:39.900+0000 7f7ca2a56700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-24T11:39:40.144 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setuser ceph since I am not root 2024-03-24T11:39:40.145 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ignoring --setgroup ceph since I am not root 2024-03-24T11:39:40.145 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:39 smithi145 ceph-mgr[25092]: -- 172.21.15.145:0/625561913 <== mon.1 v2:172.21.15.145:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559fc743e340 con 0x559fc7416800 2024-03-24T11:39:40.155 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:40.155 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' 2024-03-24T11:39:40.155 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:40.155 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-24T11:39:40.155 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: Failing over to other MGR 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-mon[25685]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:39] ENGINE Bus STOPPING 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:39 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:39] ENGINE Bus STOPPED 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:40] ENGINE Bus STARTING 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:40] ENGINE Serving on http://:::9283 2024-03-24T11:39:40.156 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:39:40] ENGINE Bus STARTED 2024-03-24T11:39:40.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:40.425+0000 7f4396817000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:39:40.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:40.519+0000 7f4396817000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:39:41.204 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.114+0000 7f4396817000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:39:41.204 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.203+0000 7f4396817000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:39:41.204 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-24T11:39:41.204 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: mgrmap e22: x(active, starting, since 0.991809s) 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:41.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: Manager daemon x is now available 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: Queued rgw.foo for migration 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: Queued rgw.smpl for migration 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:41.206 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: mgrmap e22: x(active, starting, since 0.991809s) 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:39:41.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: Manager daemon x is now available 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: Queued rgw.foo for migration 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: Queued rgw.smpl for migration 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-24T11:39:41.208 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.209 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:41.209 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:41.209 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:41.209 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:41.209 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: Deploying cephadm binary to smithi145 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[24853]: Deploying cephadm binary to smithi149 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.210 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.211 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: Deploying cephadm binary to smithi145 2024-03-24T11:39:41.211 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:40 smithi145 ceph-mon[29394]: Deploying cephadm binary to smithi149 2024-03-24T11:39:41.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.14442 172.21.15.145:0/265232208' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-24T11:39:41.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: mgrmap e22: x(active, starting, since 0.991809s) 2024-03-24T11:39:41.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:39:41.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: Manager daemon x is now available 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: Queued rgw.foo for migration 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: Queued rgw.smpl for migration 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:41.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T11:39:41.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:41.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: Deploying cephadm binary to smithi145 2024-03-24T11:39:41.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:40 smithi149 ceph-mon[25685]: Deploying cephadm binary to smithi149 2024-03-24T11:39:41.518 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.287+0000 7f4396817000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:39:41.518 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.437+0000 7f4396817000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:39:41.518 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.517+0000 7f4396817000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:39:41.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:41.544Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:39:41.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.625+0000 7f4396817000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:41.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.708+0000 7f4396817000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:39:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:41 smithi149 ceph-mon[25685]: [24/Mar/2024:11:39:40] ENGINE Bus STARTING 2024-03-24T11:39:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:41 smithi149 ceph-mon[25685]: [24/Mar/2024:11:39:40] ENGINE Serving on http://172.21.15.149:8765 2024-03-24T11:39:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:41 smithi149 ceph-mon[25685]: [24/Mar/2024:11:39:40] ENGINE Serving on https://172.21.15.149:7150 2024-03-24T11:39:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:41 smithi149 ceph-mon[25685]: [24/Mar/2024:11:39:40] ENGINE Bus STARTED 2024-03-24T11:39:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:41 smithi149 ceph-mon[25685]: mgrmap e23: x(active, since 2s) 2024-03-24T11:39:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:41 smithi149 ceph-mon[25685]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[29394]: [24/Mar/2024:11:39:40] ENGINE Bus STARTING 2024-03-24T11:39:42.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[29394]: [24/Mar/2024:11:39:40] ENGINE Serving on http://172.21.15.149:8765 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[29394]: [24/Mar/2024:11:39:40] ENGINE Serving on https://172.21.15.149:7150 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[29394]: [24/Mar/2024:11:39:40] ENGINE Bus STARTED 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[29394]: mgrmap e23: x(active, since 2s) 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[29394]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.871+0000 7f4396817000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:41.959+0000 7f4396817000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[24853]: [24/Mar/2024:11:39:40] ENGINE Bus STARTING 2024-03-24T11:39:42.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[24853]: [24/Mar/2024:11:39:40] ENGINE Serving on http://172.21.15.149:8765 2024-03-24T11:39:42.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[24853]: [24/Mar/2024:11:39:40] ENGINE Serving on https://172.21.15.149:7150 2024-03-24T11:39:42.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[24853]: [24/Mar/2024:11:39:40] ENGINE Bus STARTED 2024-03-24T11:39:42.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[24853]: mgrmap e23: x(active, since 2s) 2024-03-24T11:39:42.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:41 smithi145 ceph-mon[24853]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:43.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:42 smithi149 ceph-mon[25685]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:43.302 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:42 smithi145 ceph-mon[29394]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:43.302 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:42 smithi145 ceph-mon[24853]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:43.302 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:42.983+0000 7f4396817000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:39:43.302 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:43.112+0000 7f4396817000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:39:43.303 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:43.301+0000 7f4396817000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:39:43.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:43.388+0000 7f4396817000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:39:43.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:43.522+0000 7f4396817000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:43.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:39:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:39:43.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:39:43.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:43.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:39:44.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-mon[29394]: mgrmap e24: x(active, since 4s) 2024-03-24T11:39:44.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-mon[24853]: mgrmap e24: x(active, since 4s) 2024-03-24T11:39:44.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:43.841+0000 7f4396817000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:39:44.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:44.063+0000 7f4396817000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:39:44.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:43 smithi149 ceph-mon[25685]: mgrmap e24: x(active, since 4s) 2024-03-24T11:39:44.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:44.149+0000 7f4396817000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:39:44.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:44.317+0000 7f4396817000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:39:45.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:44 smithi149 ceph-mon[25685]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:45.254 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:44 smithi145 ceph-mon[29394]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:45.254 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:44 smithi145 ceph-mon[24853]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:45.576 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:45.253+0000 7f4396817000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:39:45.577 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:45.576+0000 7f4396817000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:39:45.849 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:45.696+0000 7f4396817000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:39:45.849 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:45.848+0000 7f4396817000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:39:46.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[24853]: Standby manager daemon y started 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[29394]: Standby manager daemon y started 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T11:39:46.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:39:46.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:39:45] ENGINE Bus STARTING 2024-03-24T11:39:46.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: CherryPy Checker: 2024-03-24T11:39:46.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: The Application mounted at '' has an empty config. 2024-03-24T11:39:46.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: 2024-03-24T11:39:46.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:39:45] ENGINE Serving on http://:::9283 2024-03-24T11:39:46.113 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: [24/Mar/2024:11:39:45] ENGINE Bus STARTED 2024-03-24T11:39:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:45 smithi149 ceph-mon[25685]: Standby manager daemon y started 2024-03-24T11:39:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:45 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T11:39:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:45 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:39:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:45 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T11:39:46.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:45 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/2200866754' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:39:47.068 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:46 smithi145 ceph-mon[29394]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:47.069 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:46 smithi145 ceph-mon[29394]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-24T11:39:47.069 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:46 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:39:47.069 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:46 smithi145 ceph-mon[24853]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:47.069 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:46 smithi145 ceph-mon[24853]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-24T11:39:47.069 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:46 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:39:47.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:46 smithi149 ceph-mon[25685]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:39:47.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:46 smithi149 ceph-mon[25685]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-24T11:39:47.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:46 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:39:47.361 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:47.068Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:48.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:47 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:48.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:47] "GET /metrics HTTP/1.1" 200 35165 "" "Prometheus/2.33.4" 2024-03-24T11:39:48.282 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:47 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:48.282 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:47 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:48 smithi149 ceph-mon[25685]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:39:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:48 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:48 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:49.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:48 smithi145 ceph-mon[24853]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:39:49.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:48 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:49.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:48 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:49.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:48 smithi145 ceph-mon[29394]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:39:49.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:48 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:49.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:48 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi145 to 2736M 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi145 to 2736M 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi145 to 2736M 2024-03-24T11:39:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:39:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:52.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:52 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:52.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:52 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:52.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:52 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:52.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:52 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:52 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:52 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:53 smithi149 ceph-mon[25685]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:39:53.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:53 smithi145 ceph-mon[24853]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:39:53.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:53 smithi145 ceph-mon[29394]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:39:53.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:39:53.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:53.863 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:39:53.863 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:39:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:39:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:39:54.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:39:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:39:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:55.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:55.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:55.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:39:55.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:55.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:55.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:56.764 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:56 smithi145 systemd[1]: Stopping Ceph node-exporter.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:39:56.764 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:56 smithi145 bash[58321]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-24T11:39:56.764 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:56 smithi145 podman[58334]: 2024-03-24 11:39:56.677039233 +0000 UTC m=+0.115434676 container died 7399120d8cc1c529a5128bdddc8e9fcc85e8e34a5af55bcea872c5920fc89335 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:56.764 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:39:56.764 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:39:56.764 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.765 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.767 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.767 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.767 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.767 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.767 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.767 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.767 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:39:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:39:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:39:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:39:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:39:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:39:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:57.435 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 podman[58334]: 2024-03-24 11:39:57.191997451 +0000 UTC m=+0.630392890 container cleanup 7399120d8cc1c529a5128bdddc8e9fcc85e8e34a5af55bcea872c5920fc89335 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:57.435 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 bash[58334]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T11:39:57.435 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 bash[58361]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-24T11:39:57.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[29394]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:39:57.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[29394]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:39:57.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[29394]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:57.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[29394]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-24T11:39:57.780 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[29394]: Reconfiguring daemon node-exporter.a on smithi145 2024-03-24T11:39:57.780 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 podman[58348]: 2024-03-24 11:39:57.435149438 +0000 UTC m=+0.758515126 container remove 7399120d8cc1c529a5128bdddc8e9fcc85e8e34a5af55bcea872c5920fc89335 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:57.780 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:39:57.780 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T11:39:57.780 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 systemd[1]: Stopped Ceph node-exporter.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:39:57.781 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 2.654s CPU time. 2024-03-24T11:39:57.781 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 systemd[1]: Starting Ceph node-exporter.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:39:57.781 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 podman[58440]: 2024-03-24 11:39:57.660976843 +0000 UTC m=+0.111776064 container create 77953b6c5fa5f0c08cf66fbd4f7174f6b95360235920cc548d36f83afa388c4a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:57.781 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 podman[58440]: 2024-03-24 11:39:57.572825348 +0000 UTC m=+0.023624569 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-24T11:39:57.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[24853]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:39:57.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[24853]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:39:57.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[24853]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:57.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[24853]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-24T11:39:57.781 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-mon[24853]: Reconfiguring daemon node-exporter.a on smithi145 2024-03-24T11:39:57.792 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:57 smithi149 ceph-mon[25685]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:39:57.792 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:57 smithi149 ceph-mon[25685]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:39:57.792 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:57 smithi149 ceph-mon[25685]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:57.792 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:57 smithi149 ceph-mon[25685]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-24T11:39:57.792 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:57 smithi149 ceph-mon[25685]: Reconfiguring daemon node-exporter.a on smithi145 2024-03-24T11:39:58.111 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 podman[58440]: 2024-03-24 11:39:57.855224382 +0000 UTC m=+0.306023612 container init 77953b6c5fa5f0c08cf66fbd4f7174f6b95360235920cc548d36f83afa388c4a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 podman[58440]: 2024-03-24 11:39:57.857967164 +0000 UTC m=+0.308766396 container start 77953b6c5fa5f0c08cf66fbd4f7174f6b95360235920cc548d36f83afa388c4a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.861Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.861Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.861Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.861Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=arp 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-24T11:39:58.112 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-24T11:39:58.113 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-24T11:39:58.113 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-24T11:39:58.113 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=edac 2024-03-24T11:39:58.113 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-24T11:39:58.113 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-24T11:39:58.113 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-24T11:39:58.113 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=os 2024-03-24T11:39:58.114 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=stat 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=time 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=uname 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-24T11:39:58.115 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-24T11:39:58.116 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.862Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-24T11:39:58.116 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.863Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-24T11:39:58.116 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a[58456]: ts=2024-03-24T11:39:57.863Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-24T11:39:58.116 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 bash[58440]: 77953b6c5fa5f0c08cf66fbd4f7174f6b95360235920cc548d36f83afa388c4a 2024-03-24T11:39:58.116 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:39:57 smithi145 systemd[1]: Started Ceph node-exporter.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:39:58.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:39:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:39:57] "GET /metrics HTTP/1.1" 200 38158 "" "Prometheus/2.33.4" 2024-03-24T11:39:58.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:58 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:58.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:58 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:58.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:58 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:58.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:58 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:58.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:58 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:58.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:58 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:58 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:58 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:58 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:39:59.135 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:39:59.817 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (4m) 11s ago 4m 28.2M - 0.23.0 ba2b418f427c b0ee65c327d9 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (4m) 9s ago 4m 53.5M - 8.3.5 dad864ee21e9 1ea96d14b839 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (3m) 11s ago 3m 53.2M - 3.5 e1d6a67b021e 149865107de4 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283 running (36s) 9s ago 9m 496M - 19.0.0-1667-gdb0330b1 4f325f8440e9 25419047a080 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:9283 running (10m) 11s ago 10m 385M - 17.2.0 e1d6a67b021e 642048ee9b24 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (10m) 11s ago 11m 50.5M 2048M 17.2.0 e1d6a67b021e ca31a24577e6 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (9m) 9s ago 9m 40.2M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (9m) 11s ago 9m 40.3M 2048M 17.2.0 e1d6a67b021e 0b79dbdf9a4c 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 starting - - - - 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (5m) 9s ago 5m 16.4M - 1.3.1 1dbe0e931976 edb4947ca62a 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (8m) 11s ago 8m 51.0M 2736M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (8m) 11s ago 8m 52.7M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:39:59.818 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (8m) 11s ago 8m 48.3M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (7m) 11s ago 7m 49.9M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (7m) 9s ago 7m 53.5M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (7m) 9s ago 7m 49.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (6m) 9s ago 6m 47.6M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (6m) 9s ago 6m 50.4M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (4m) 9s ago 5m 64.7M - 2.33.4 514e6a882f6e b35d6bc7db17 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (4m) 11s ago 3m 69.3M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (3m) 9s ago 3m 73.2M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (3m) 11s ago 3m 70.7M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:39:59.819 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (3m) 9s ago 3m 71.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:39:59.819 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:59 smithi145 ceph-mon[29394]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:59.820 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:59 smithi145 ceph-mon[29394]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:39:59.820 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:39:59 smithi145 ceph-mon[29394]: Reconfiguring daemon alertmanager.a on smithi145 2024-03-24T11:39:59.820 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:59 smithi145 ceph-mon[24853]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:59.820 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:59 smithi145 ceph-mon[24853]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:39:59.820 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:39:59 smithi145 ceph-mon[24853]: Reconfiguring daemon alertmanager.a on smithi145 2024-03-24T11:39:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:59 smithi149 ceph-mon[25685]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:39:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:59 smithi149 ceph-mon[25685]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:39:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:39:59 smithi149 ceph-mon[25685]: Reconfiguring daemon alertmanager.a on smithi145 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:40:00.244 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": {}, 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:40:00.245 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:40:00.582 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:40:00.582 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:40:00.582 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:40:00.582 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:40:00.582 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [], 2024-03-24T11:40:00.582 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T11:40:00.583 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "", 2024-03-24T11:40:00.583 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:40:00.583 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:40:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[29394]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[29394]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[29394]: overall HEALTH_OK 2024-03-24T11:40:00.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1860831351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:40:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[24853]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[24853]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[24853]: overall HEALTH_OK 2024-03-24T11:40:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:00 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1860831351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:40:00.876 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:00 smithi149 ceph-mon[25685]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:00.877 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:00 smithi149 ceph-mon[25685]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:00.877 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:00 smithi149 ceph-mon[25685]: overall HEALTH_OK 2024-03-24T11:40:00.877 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:00 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1860831351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:40:00.977 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:40:01.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[24853]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:01.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[24853]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:01.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[24853]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:01.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/396409264' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:40:01.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[29394]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:01.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[29394]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:01.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[29394]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:01.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:01 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/396409264' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:40:01.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:01 smithi149 ceph-mon[25685]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:01.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:01 smithi149 ceph-mon[25685]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:01.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:01 smithi149 ceph-mon[25685]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:01.888 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:01 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/396409264' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:40:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:02 smithi149 ceph-mon[25685]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:02.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.145:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:40:02.817 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:02 smithi145 ceph-mon[29394]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:02.817 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:02 smithi145 ceph-mon[24853]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:03.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=error ts=2024-03-24T11:40:03.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:40:03.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:40:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.145:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.145 because it doesn't contain any IP SANs" 2024-03-24T11:40:03.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=warn ts=2024-03-24T11:40:03.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-03-24T11:40:04.225 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:40:05.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:04 smithi149 ceph-mon[25685]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:05.321 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:04 smithi145 ceph-mon[24853]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:05.321 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:04 smithi145 ceph-mon[29394]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:05.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:05 smithi145 systemd[1]: Stopping Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:05.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:05 smithi145 bash[58855]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-24T11:40:05.972 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[49422]: level=info ts=2024-03-24T11:40:05.687Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:40:05.972 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:05 smithi145 podman[58868]: 2024-03-24 11:40:05.707746237 +0000 UTC m=+0.147584757 container died b0ee65c327d9c3ea9990ced7d93234b2baf1f1a92484483fbb513cd34fe5b9a5 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:06.330 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 podman[58868]: 2024-03-24 11:40:06.062145498 +0000 UTC m=+0.501984016 container cleanup b0ee65c327d9c3ea9990ced7d93234b2baf1f1a92484483fbb513cd34fe5b9a5 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:06.331 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 bash[58868]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:40:06.331 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 bash[58896]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-24T11:40:06.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 podman[58882]: 2024-03-24 11:40:06.330433953 +0000 UTC m=+0.621986015 container remove b0ee65c327d9c3ea9990ced7d93234b2baf1f1a92484483fbb513cd34fe5b9a5 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:06.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 podman[58882]: 2024-03-24 11:40:06.414506921 +0000 UTC m=+0.706058932 volume remove 1a5be6e0324a9120481fbaa25af585279a8623f7a6f42a5f0598c276f2dde5a0 2024-03-24T11:40:06.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:40:06.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 systemd[1]: Stopped Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:06.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 systemd[1]: Starting Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:06.926 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 podman[58968]: 2024-03-24 11:40:06.631960374 +0000 UTC m=+0.094947187 volume create 7a54861f36cc75142d3cbdc6e672a80bf4789b1ba02e3ed1fd39814fa98706a9 2024-03-24T11:40:06.927 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 podman[58968]: 2024-03-24 11:40:06.555302725 +0000 UTC m=+0.018289558 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-24T11:40:06.927 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:06 smithi145 podman[58968]: 2024-03-24 11:40:06.758199784 +0000 UTC m=+0.221186639 container create e32a8e22dad4d60a93060528f3b6c6d6513623a23c5a11a7bcfc383f578f0eb4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:07.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:06 smithi149 ceph-mon[25685]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:07.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:06 smithi145 ceph-mon[29394]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:07.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:06 smithi145 ceph-mon[24853]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 podman[58968]: 2024-03-24 11:40:07.069813687 +0000 UTC m=+0.532800525 container init e32a8e22dad4d60a93060528f3b6c6d6513623a23c5a11a7bcfc383f578f0eb4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 podman[58968]: 2024-03-24 11:40:07.072579922 +0000 UTC m=+0.535566745 container start e32a8e22dad4d60a93060528f3b6c6d6513623a23c5a11a7bcfc383f578f0eb4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.082Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.082Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.084Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.145 port=9094 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.084Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.111Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.112Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.114Z caller=main.go:518 msg=Listening address=:9093 2024-03-24T11:40:07.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:07.114Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-24T11:40:07.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 bash[58968]: e32a8e22dad4d60a93060528f3b6c6d6513623a23c5a11a7bcfc383f578f0eb4 2024-03-24T11:40:07.363 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:07 smithi145 systemd[1]: Started Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:08.197 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:07] "GET /metrics HTTP/1.1" 200 38158 "" "Prometheus/2.33.4" 2024-03-24T11:40:08.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: Reconfiguring iscsi.foo.smithi145.hnqhcg (dependencies changed)... 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: Reconfiguring daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[29394]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:08.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:08.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:08.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: Reconfiguring iscsi.foo.smithi145.hnqhcg (dependencies changed)... 2024-03-24T11:40:08.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:40:08.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:40:08.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:08.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: Reconfiguring daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:40:08.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:08 smithi145 ceph-mon[24853]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: Reconfiguring iscsi.foo.smithi145.hnqhcg (dependencies changed)... 2024-03-24T11:40:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:40:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:40:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:08.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: Reconfiguring daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:40:08.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:08 smithi149 ceph-mon[25685]: from='client.24689 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:09.357 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:09.085Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000088799s 2024-03-24T11:40:09.357 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:09 smithi145 ceph-mon[24853]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:09.357 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:09 smithi145 ceph-mon[29394]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:09.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:09 smithi149 ceph-mon[25685]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:10.268 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:10 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:10.269 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:10 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:10.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:10.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:40:10.597+0000 7feed119d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:10.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Creating ceph-iscsi config... 2024-03-24T11:40:10.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:10.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Traceback (most recent call last): 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return _run_code(code, main_globals, None, 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: exec(code, run_globals) 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:10.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:10.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:10.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:10.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:10.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Traceback (most recent call last): 2024-03-24T11:40:10.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return future.result(timeout) 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return self.__get_result() 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: raise self._exception 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: config_blobs=daemon_spec.final_config, 2024-03-24T11:40:10.978 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Creating ceph-iscsi config... 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Traceback (most recent call last): 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return _run_code(code, main_globals, None, 2024-03-24T11:40:10.979 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: exec(code, run_globals) 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:10.980 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:11 smithi149 ceph-mon[25685]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:11.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:11 smithi145 ceph-mon[29394]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:11.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:11 smithi145 ceph-mon[24853]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:12.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.145:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:40:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:12 smithi149 ceph-mon[25685]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-03-24T11:40:12.476 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:40:12.477 INFO:journalctl@ceph.mon.b.smithi149.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: return future.result(timeout) 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.__get_result() 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: raise self._exception 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: config_blobs=daemon_spec.final_config, 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:40:12.478 INFO:journalctl@ceph.mon.b.smithi149.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:12.479 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:12 smithi149 ceph-mon[25685]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:40:12.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:12 smithi149 ceph-mon[25685]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:12 smithi145 ceph-mon[24853]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.612 INFO:journalctl@ceph.mon.a.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.613 INFO:journalctl@ceph.mon.a.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: return future.result(timeout) 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: return self.__get_result() 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: raise self._exception 2024-03-24T11:40:12.614 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: config_blobs=daemon_spec.final_config, 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:40:12.615 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:12 smithi145 ceph-mon[24853]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:40:12.616 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:12 smithi145 ceph-mon[24853]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:12 smithi145 ceph-mon[29394]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.617 INFO:journalctl@ceph.mon.c.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:12.618 INFO:journalctl@ceph.mon.c.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.619 INFO:journalctl@ceph.mon.c.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.620 INFO:journalctl@ceph.mon.c.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.620 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:40:12.620 INFO:journalctl@ceph.mon.c.smithi145.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:40:12.620 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:40:12.620 INFO:journalctl@ceph.mon.c.smithi145.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: return future.result(timeout) 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: return self.__get_result() 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: raise self._exception 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: config_blobs=daemon_spec.final_config, 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:40:12.621 INFO:journalctl@ceph.mon.c.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:40:12.622 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:12 smithi145 ceph-mon[29394]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:40:12.623 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:12 smithi145 ceph-mon[29394]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:40:13.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:13 smithi149 ceph-mon[25685]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:13.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:13 smithi145 ceph-mon[29394]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:13.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:13 smithi145 ceph-mon[24853]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:14.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:40:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:15 smithi149 ceph-mon[25685]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:15.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:15 smithi145 ceph-mon[24853]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:15.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:15 smithi145 ceph-mon[29394]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:17.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:17 smithi145 ceph-mon[24853]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:17.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:17 smithi145 ceph-mon[29394]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:17.362 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:40:17 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:40:17.088Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003146853s 2024-03-24T11:40:17.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:17 smithi149 ceph-mon[25685]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:18.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:17] "GET /metrics HTTP/1.1" 200 38157 "" "Prometheus/2.33.4" 2024-03-24T11:40:19.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:18 smithi149 ceph-mon[25685]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:19.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:18 smithi145 ceph-mon[29394]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:19.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:18 smithi145 ceph-mon[24853]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:40:21.205 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:20 smithi145 ceph-mon[24853]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:40:21.205 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:20 smithi145 ceph-mon[29394]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:40:21.224 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:20 smithi149 ceph-mon[25685]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:40:22.607 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:22 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.145:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:40:23.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:22 smithi149 ceph-mon[25685]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:40:23.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:22 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2626824760' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:40:23.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:22 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1203751872' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]: dispatch 2024-03-24T11:40:23.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:22 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]: dispatch 2024-03-24T11:40:23.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[29394]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:40:23.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2626824760' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:40:23.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1203751872' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]: dispatch 2024-03-24T11:40:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]: dispatch 2024-03-24T11:40:23.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[24853]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:40:23.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2626824760' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:40:23.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1203751872' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]: dispatch 2024-03-24T11:40:23.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:22 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]: dispatch 2024-03-24T11:40:24.213 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 systemd[1]: Stopping Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:24.213 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 bash[50441]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-24T11:40:24.213 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:23 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]': finished 2024-03-24T11:40:24.213 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:23 smithi149 ceph-mon[25685]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:40:24.214 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:23 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3182479930' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]: dispatch 2024-03-24T11:40:24.214 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:23 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]: dispatch 2024-03-24T11:40:24.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]': finished 2024-03-24T11:40:24.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[29394]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:40:24.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3182479930' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]: dispatch 2024-03-24T11:40:24.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]: dispatch 2024-03-24T11:40:24.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[25088]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:40:24.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/2169988987"}]': finished 2024-03-24T11:40:24.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[24853]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:40:24.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3182479930' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]: dispatch 2024-03-24T11:40:24.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:23 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]: dispatch 2024-03-24T11:40:24.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.212Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:40:24.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:40:24.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:40:24.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-24T11:40:24.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[42601]: ts=2024-03-24T11:40:24.213Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-24T11:40:24.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 podman[50452]: 2024-03-24 11:40:24.223102518 +0000 UTC m=+0.116088040 container died b35d6bc7db17ca524016cc019c00c8df83fa4798f0cbe44c0dc8fcb8f294c82c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:24.864 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 podman[50452]: 2024-03-24 11:40:24.654796285 +0000 UTC m=+0.547781807 container cleanup b35d6bc7db17ca524016cc019c00c8df83fa4798f0cbe44c0dc8fcb8f294c82c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:24.864 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 bash[50452]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:40:25.126 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-mon[25685]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:25.126 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]': finished 2024-03-24T11:40:25.126 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-mon[25685]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:40:25.126 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2683932834' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]: dispatch 2024-03-24T11:40:25.126 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:24 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]: dispatch 2024-03-24T11:40:25.126 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 podman[50466]: 2024-03-24 11:40:24.86443631 +0000 UTC m=+0.641353618 container remove b35d6bc7db17ca524016cc019c00c8df83fa4798f0cbe44c0dc8fcb8f294c82c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:25.126 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 bash[50477]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-24T11:40:25.126 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T11:40:25.127 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 systemd[1]: Stopped Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:25.127 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.483s CPU time. 2024-03-24T11:40:25.127 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:24 smithi149 systemd[1]: Starting Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[24853]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]': finished 2024-03-24T11:40:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[24853]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:40:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2683932834' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]: dispatch 2024-03-24T11:40:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]: dispatch 2024-03-24T11:40:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[29394]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2743538076"}]': finished 2024-03-24T11:40:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[29394]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:40:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2683932834' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]: dispatch 2024-03-24T11:40:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:24 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]: dispatch 2024-03-24T11:40:25.426 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 podman[50550]: 2024-03-24 11:40:25.030623227 +0000 UTC m=+0.020564446 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-24T11:40:25.426 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 podman[50550]: 2024-03-24 11:40:25.148781258 +0000 UTC m=+0.138722456 container create 5a4dec45ed4b5659e912831994a1026d31cded5c2ee0bb862767f74bf92b0dfd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:25.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 podman[50550]: 2024-03-24 11:40:25.426999933 +0000 UTC m=+0.416941144 container init 5a4dec45ed4b5659e912831994a1026d31cded5c2ee0bb862767f74bf92b0dfd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:25.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 podman[50550]: 2024-03-24 11:40:25.42980813 +0000 UTC m=+0.419749331 container start 5a4dec45ed4b5659e912831994a1026d31cded5c2ee0bb862767f74bf92b0dfd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.453Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.453Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.453Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.453Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi149 (none))" 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.453Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.453Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.455Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.455Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.458Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.460Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:40:25.726 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.460Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.927µs 2024-03-24T11:40:25.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.460Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:40:25.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 bash[50550]: 5a4dec45ed4b5659e912831994a1026d31cded5c2ee0bb862767f74bf92b0dfd 2024-03-24T11:40:25.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 systemd[1]: Started Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:25.727 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.658Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-24T11:40:26.215 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 systemd[1]: Stopping Ceph node-exporter.b for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:26.215 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 bash[50673]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-24T11:40:26.216 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]': finished 2024-03-24T11:40:26.216 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-mon[25685]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:40:26.216 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:26.216 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2941365597' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]: dispatch 2024-03-24T11:40:26.216 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]: dispatch 2024-03-24T11:40:26.216 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:26.216 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:26.216 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.914Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-24T11:40:26.217 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.914Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-24T11:40:26.217 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.914Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.11µs wal_replay_duration=454.048451ms total_replay_duration=454.093073ms 2024-03-24T11:40:26.217 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.917Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:40:26.217 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.917Z caller=main.go:947 level=info msg="TSDB started" 2024-03-24T11:40:26.217 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.917Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:40:26.217 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.942Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.124851ms db_storage=742ns remote_storage=1.562µs web_handler=752ns query_engine=681ns scrape=432.532µs scrape_sd=154.312µs notify=16.775µs notify_sd=236.933µs rules=23.036069ms 2024-03-24T11:40:26.217 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:25.942Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]': finished 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[29394]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2941365597' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]: dispatch 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]: dispatch 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:26.263 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/2169988987"}]': finished 2024-03-24T11:40:26.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[24853]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:40:26.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:26.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2941365597' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]: dispatch 2024-03-24T11:40:26.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]: dispatch 2024-03-24T11:40:26.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:26.264 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:25 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:26.475 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 podman[50684]: 2024-03-24 11:40:26.215543205 +0000 UTC m=+0.118980692 container died edb4947ca62a6b7151d17a1ef2f75bed13f440405471da77a36a9b6064101fab (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:26.857 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 podman[50684]: 2024-03-24 11:40:26.630594811 +0000 UTC m=+0.534032312 container cleanup edb4947ca62a6b7151d17a1ef2f75bed13f440405471da77a36a9b6064101fab (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:26.857 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 bash[50684]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T11:40:26.857 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 bash[50711]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-24T11:40:27.124 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:26 smithi149 ceph-mon[25685]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-24T11:40:27.124 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:26 smithi149 ceph-mon[25685]: Reconfiguring daemon node-exporter.b on smithi149 2024-03-24T11:40:27.124 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:26 smithi149 ceph-mon[25685]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:27.124 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:26 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]': finished 2024-03-24T11:40:27.124 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:26 smithi149 ceph-mon[25685]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:40:27.124 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:26 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1386466229' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]: dispatch 2024-03-24T11:40:27.125 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:26 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]: dispatch 2024-03-24T11:40:27.125 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 podman[50698]: 2024-03-24 11:40:26.890267907 +0000 UTC m=+0.675322495 container remove edb4947ca62a6b7151d17a1ef2f75bed13f440405471da77a36a9b6064101fab (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:27.125 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:40:27.125 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T11:40:27.125 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 systemd[1]: Stopped Ceph node-exporter.b for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:27.125 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 2.754s CPU time. 2024-03-24T11:40:27.125 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:26 smithi149 systemd[1]: Starting Ceph node-exporter.b for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:27.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[24853]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[24853]: Reconfiguring daemon node-exporter.b on smithi149 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[24853]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]': finished 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[24853]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1386466229' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]: dispatch 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]: dispatch 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[29394]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[29394]: Reconfiguring daemon node-exporter.b on smithi149 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[29394]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:40:27.278 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2864908609"}]': finished 2024-03-24T11:40:27.279 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[29394]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:40:27.279 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1386466229' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]: dispatch 2024-03-24T11:40:27.279 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:26 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]: dispatch 2024-03-24T11:40:27.477 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:27 smithi149 podman[50787]: 2024-03-24 11:40:27.028204717 +0000 UTC m=+0.020134690 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-24T11:40:27.966 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:27 smithi149 podman[50787]: 2024-03-24 11:40:27.777836584 +0000 UTC m=+0.769766527 container create 54579c2f54d5e391ab02aa731a6a769dd07e7cd0eee132433a7d5e650da44ddd (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:27 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]': finished 2024-03-24T11:40:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:27 smithi149 ceph-mon[25685]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:40:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:27 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2798390896' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]: dispatch 2024-03-24T11:40:28.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:27 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]: dispatch 2024-03-24T11:40:28.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]': finished 2024-03-24T11:40:28.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[29394]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:40:28.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2798390896' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]: dispatch 2024-03-24T11:40:28.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]: dispatch 2024-03-24T11:40:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2920519761"}]': finished 2024-03-24T11:40:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[24853]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:40:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2798390896' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]: dispatch 2024-03-24T11:40:28.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:27 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]: dispatch 2024-03-24T11:40:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-mon[25685]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:40:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]': finished 2024-03-24T11:40:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-mon[25685]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:40:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3519689256' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]: dispatch 2024-03-24T11:40:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]: dispatch 2024-03-24T11:40:29.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 podman[50787]: 2024-03-24 11:40:28.923291427 +0000 UTC m=+1.915221383 container init 54579c2f54d5e391ab02aa731a6a769dd07e7cd0eee132433a7d5e650da44ddd (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:29.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 podman[50787]: 2024-03-24 11:40:28.926834958 +0000 UTC m=+1.918764901 container start 54579c2f54d5e391ab02aa731a6a769dd07e7cd0eee132433a7d5e650da44ddd (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:40:29.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.930Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-24T11:40:29.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.930Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-24T11:40:29.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.931Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-24T11:40:29.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.931Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:40:29.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=arp 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=edac 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-24T11:40:29.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=os 2024-03-24T11:40:29.228 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-24T11:40:29.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-24T11:40:29.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-24T11:40:29.229 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=stat 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=time 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=uname 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-24T11:40:29.230 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-24T11:40:29.231 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-24T11:40:29.231 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b[50804]: ts=2024-03-24T11:40:28.932Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-24T11:40:29.231 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:29 smithi149 bash[50787]: 54579c2f54d5e391ab02aa731a6a769dd07e7cd0eee132433a7d5e650da44ddd 2024-03-24T11:40:29.231 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:40:29 smithi149 systemd[1]: Started Ceph node-exporter.b for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:29.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[29394]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:40:29.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]': finished 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[29394]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3519689256' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]: dispatch 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]: dispatch 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[24853]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3932693267"}]': finished 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[24853]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3519689256' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]: dispatch 2024-03-24T11:40:29.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:28 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]: dispatch 2024-03-24T11:40:30.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]': finished 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: Reconfiguring grafana.a (dependencies changed)... 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[24853]: Reconfiguring daemon grafana.a on smithi149 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]': finished 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: Reconfiguring grafana.a (dependencies changed)... 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:29 smithi145 ceph-mon[29394]: Reconfiguring daemon grafana.a on smithi149 2024-03-24T11:40:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1874048018"}]': finished 2024-03-24T11:40:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:40:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: Reconfiguring grafana.a (dependencies changed)... 2024-03-24T11:40:30.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:40:30.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:30.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:40:30.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:29 smithi149 ceph-mon[25685]: Reconfiguring daemon grafana.a on smithi149 2024-03-24T11:40:31.331 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:40:31.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:31 smithi149 ceph-mon[25685]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:40:31.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:31 smithi145 ceph-mon[24853]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:40:31.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:31 smithi145 ceph-mon[29394]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-24T11:40:31.966 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:40:31.966 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 starting - - - - 2024-03-24T11:40:31.966 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (5m) 41s ago 5m 53.5M - 8.3.5 dad864ee21e9 1ea96d14b839 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (4m) 43s ago 4m 53.2M - 3.5 e1d6a67b021e 149865107de4 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283 running (68s) 41s ago 9m 496M - 19.0.0-1667-gdb0330b1 4f325f8440e9 25419047a080 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:9283 running (11m) 43s ago 11m 385M - 17.2.0 e1d6a67b021e 642048ee9b24 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (11m) 43s ago 11m 50.5M 2048M 17.2.0 e1d6a67b021e ca31a24577e6 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (10m) 41s ago 10m 40.2M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (10m) 43s ago 10m 40.3M 2048M 17.2.0 e1d6a67b021e 0b79dbdf9a4c 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 starting - - - - 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 starting - - - - 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (9m) 43s ago 9m 51.0M 2736M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (9m) 43s ago 9m 52.7M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (8m) 43s ago 8m 48.3M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (8m) 43s ago 8m 49.9M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (7m) 41s ago 7m 53.5M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:40:31.967 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (7m) 41s ago 7m 49.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:40:31.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (7m) 41s ago 7m 47.6M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:40:31.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (6m) 41s ago 6m 50.4M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:40:31.968 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 starting - - - - 2024-03-24T11:40:31.968 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (4m) 43s ago 4m 69.3M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:40:31.968 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (4m) 41s ago 4m 73.2M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:40:31.968 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (4m) 43s ago 4m 70.7M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:40:31.968 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (4m) 41s ago 4m 71.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:40:32.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": {}, 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:40:32.356 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:40:32.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:32 smithi149 ceph-mon[25685]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:32.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:32 smithi145 ceph-mon[24853]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:32.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:32 smithi145 ceph-mon[29394]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:32.682 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:40:32.682 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:40:32.682 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:40:32.682 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:40:32.683 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [], 2024-03-24T11:40:32.683 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "1/23 daemons upgraded", 2024-03-24T11:40:32.683 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "", 2024-03-24T11:40:32.683 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:40:32.683 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:40:33.066 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:40:33.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[29394]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:33.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[29394]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[29394]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[29394]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3520627052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/527765912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[24853]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[24853]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[24853]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[24853]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:33.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3520627052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:40:33.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:33 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/527765912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:40:33.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:33 smithi149 ceph-mon[25685]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:33.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:33 smithi149 ceph-mon[25685]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-24T11:40:33.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:33 smithi149 ceph-mon[25685]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:33.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:33 smithi149 ceph-mon[25685]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:33.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:33 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3520627052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:40:33.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:33 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/527765912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:40:34.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:34 smithi149 ceph-mon[25685]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:34.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:34 smithi145 ceph-mon[24853]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:34.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:34 smithi145 ceph-mon[29394]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:40:35.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:35 smithi149 ceph-mon[25685]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2024-03-24T11:40:35.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:35 smithi145 ceph-mon[29394]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2024-03-24T11:40:35.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:35 smithi145 ceph-mon[24853]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2024-03-24T11:40:37.206 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:36 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:36] "GET /metrics HTTP/1.1" 200 38163 "" "Prometheus/2.33.4" 2024-03-24T11:40:37.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:37 smithi149 ceph-mon[25685]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:37.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:37 smithi145 ceph-mon[29394]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:37.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:37 smithi145 ceph-mon[24853]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:38.931 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:38 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:40:39.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:38 smithi149 ceph-mon[25685]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:40:39.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:38 smithi145 ceph-mon[29394]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:40:39.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:38 smithi145 ceph-mon[24853]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:40:41.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:40 smithi149 ceph-mon[25685]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-24T11:40:41.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:40 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:41.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:40 smithi145 ceph-mon[29394]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-24T11:40:41.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:40 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:41.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:40 smithi145 ceph-mon[24853]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-24T11:40:41.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:40 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:42.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:42 smithi149 systemd[1]: Stopping Ceph grafana.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:42.726 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:42 smithi149 bash[51004]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-24T11:40:42.726 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:42 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[41876]: t=2024-03-24T11:40:42+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-24T11:40:42.726 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:42 smithi149 podman[51018]: 2024-03-24 11:40:42.592870415 +0000 UTC m=+0.137754398 container died 1ea96d14b839a799485bdd84db2d44a594635a334028f3d00383c303b13e91de (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.openshift.expose-services=, distribution-scope=public, maintainer=Paul Cuzner , version=8.5, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.component=ubi8-container, release=236.1648460182, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container) 2024-03-24T11:40:43.208 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:42 smithi149 ceph-mon[25685]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:43.208 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:42 smithi149 ceph-mon[25685]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:43.209 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:42 smithi149 podman[51018]: 2024-03-24 11:40:42.99100849 +0000 UTC m=+0.535892483 container cleanup 1ea96d14b839a799485bdd84db2d44a594635a334028f3d00383c303b13e91de (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, description=Ceph Grafana Container, distribution-scope=public, maintainer=Paul Cuzner , release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, name=ubi8, com.redhat.component=ubi8-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-24T11:40:43.209 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:42 smithi149 bash[51018]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a 2024-03-24T11:40:43.209 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 bash[51045]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-24T11:40:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:42 smithi145 ceph-mon[29394]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:42 smithi145 ceph-mon[29394]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:42 smithi145 ceph-mon[24853]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:43.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:42 smithi145 ceph-mon[24853]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:43.475 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 podman[51032]: 2024-03-24 11:40:43.208486352 +0000 UTC m=+0.615590781 container remove 1ea96d14b839a799485bdd84db2d44a594635a334028f3d00383c303b13e91de (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, version=8.5, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc.) 2024-03-24T11:40:43.476 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-24T11:40:43.476 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 systemd[1]: Stopped Ceph grafana.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:43.476 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@grafana.a.service: Consumed 2.624s CPU time. 2024-03-24T11:40:43.476 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 systemd[1]: Starting Ceph grafana.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:43.476 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 podman[51117]: 2024-03-24 11:40:43.34165799 +0000 UTC m=+0.017532209 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-24T11:40:43.742 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 podman[51117]: 2024-03-24 11:40:43.493002907 +0000 UTC m=+0.168877126 container create 1465ddad3f0c83203b96c92fda302639c46621dff9f6f4cd3bdd1f1981d34ab7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, description=Ceph Grafana Container, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, distribution-scope=public, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, maintainer=Paul Cuzner , io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-24T11:40:43.743 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 podman[51117]: 2024-03-24 11:40:43.695905381 +0000 UTC m=+0.371779591 container init 1465ddad3f0c83203b96c92fda302639c46621dff9f6f4cd3bdd1f1981d34ab7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, release=236.1648460182, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, description=Ceph Grafana Container, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-24T11:40:43.743 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 podman[51117]: 2024-03-24 11:40:43.698984073 +0000 UTC m=+0.374858286 container start 1465ddad3f0c83203b96c92fda302639c46621dff9f6f4cd3bdd1f1981d34ab7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, version=8.5, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, vendor=Red Hat, Inc., vcs-type=git, io.buildah.version=1.24.2) 2024-03-24T11:40:43.743 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-24T11:40:43.743 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-24T11:40:43.743 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="App mode production" logger=settings 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-24T11:40:43.744 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T11:40:43.745 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-24T11:40:43.745 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=535.039µs 2024-03-24T11:40:43.745 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-24T11:40:43.745 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-24T11:40:44.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-24T11:40:44.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 bash[51117]: 1465ddad3f0c83203b96c92fda302639c46621dff9f6f4cd3bdd1f1981d34ab7 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 systemd[1]: Started Ceph grafana.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T11:40:44.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:40:43+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-24T11:40:44.087 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Bus STOPPING 2024-03-24T11:40:44.087 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Bus STOPPED 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Bus STARTING 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Serving on http://:::9283 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Bus STARTED 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Bus STOPPING 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Bus STOPPED 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:43] ENGINE Bus STARTING 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE Serving on http://:::9283 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE Bus STARTED 2024-03-24T11:40:44.088 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE Bus STOPPING 2024-03-24T11:40:44.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:40:44.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE Bus STOPPED 2024-03-24T11:40:44.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE Bus STARTING 2024-03-24T11:40:44.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE Serving on http://:::9283 2024-03-24T11:40:44.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:40:44] ENGINE Bus STARTED 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi145.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi145.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:45.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[29394]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi145.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi145.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:40:45.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:40:45.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:40:45.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:44 smithi145 ceph-mon[24853]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:45.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi145.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi145.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:40:45.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:40:45.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:40:45.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:45.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:45.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:40:45.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:40:45.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:44 smithi149 ceph-mon[25685]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:46.754 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[29394]: Upgrade: Updating mgr.y 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[24853]: Upgrade: Updating mgr.y 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:40:46.755 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:40:46.756 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:46 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:46.785 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:46 smithi149 ceph-mon[25685]: Upgrade: Updating mgr.y 2024-03-24T11:40:46.785 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:46 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:46.785 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:46 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:40:46.785 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:46 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:40:46.785 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:46 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:40:46.785 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:46 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:47.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:46] "GET /metrics HTTP/1.1" 200 38153 "" "Prometheus/2.33.4" 2024-03-24T11:40:47.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:47 smithi145 ceph-mon[24853]: Deploying daemon mgr.y on smithi145 2024-03-24T11:40:47.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:47 smithi145 ceph-mon[24853]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:47.612 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:47 smithi145 systemd[1]: Stopping Ceph mgr.y for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:47.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:47 smithi145 ceph-mon[29394]: Deploying daemon mgr.y on smithi145 2024-03-24T11:40:47.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:47 smithi145 ceph-mon[29394]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:47.910 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:47 smithi145 podman[60081]: 2024-03-24 11:40:47.632651198 +0000 UTC m=+0.157122788 container died 642048ee9b24c21d939cdbd7e0596ba05fe8e9727d98789589b1b1d36c04fcc3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, io.openshift.expose-services=, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, name=centos-stream, release=754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_CLEAN=True, ceph=True) 2024-03-24T11:40:47.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:47 smithi149 ceph-mon[25685]: Deploying daemon mgr.y on smithi145 2024-03-24T11:40:47.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:47 smithi149 ceph-mon[25685]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:48.344 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 podman[60081]: 2024-03-24 11:40:48.084928641 +0000 UTC m=+0.609400303 container cleanup 642048ee9b24c21d939cdbd7e0596ba05fe8e9727d98789589b1b1d36c04fcc3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD) 2024-03-24T11:40:48.345 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 bash[60081]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y 2024-03-24T11:40:48.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 podman[60095]: 2024-03-24 11:40:48.344783977 +0000 UTC m=+0.711301115 container remove 642048ee9b24c21d939cdbd7e0596ba05fe8e9727d98789589b1b1d36c04fcc3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, distribution-scope=public, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=) 2024-03-24T11:40:48.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:40:48.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-24T11:40:48.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 systemd[1]: Stopped Ceph mgr.y for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:48.612 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service: Consumed 53.733s CPU time. 2024-03-24T11:40:48.931 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:40:49.023 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:48 smithi145 ceph-mon[24853]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:49.024 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:48 smithi145 ceph-mon[29394]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:49.024 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 systemd[1]: Starting Ceph mgr.y for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:40:49.024 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 podman[60221]: 2024-03-24 11:40:48.806946597 +0000 UTC m=+0.018817446 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:40:49.024 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:48 smithi145 podman[60221]: 2024-03-24 11:40:48.903787655 +0000 UTC m=+0.115658499 container create 387fa4d011fea73b0e1e11d44a57022473cea2a2bd2cd6ee257e60b5cc0b372d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T11:40:49.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:48 smithi149 ceph-mon[25685]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:49.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:49 smithi145 podman[60221]: 2024-03-24 11:40:49.156445592 +0000 UTC m=+0.368316455 container init 387fa4d011fea73b0e1e11d44a57022473cea2a2bd2cd6ee257e60b5cc0b372d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T11:40:49.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:49 smithi145 podman[60221]: 2024-03-24 11:40:49.159095116 +0000 UTC m=+0.370965961 container start 387fa4d011fea73b0e1e11d44a57022473cea2a2bd2cd6ee257e60b5cc0b372d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:40:49.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:49 smithi145 ceph-mgr[60239]: -- 172.21.15.145:0/3629383583 <== mon.1 v2:172.21.15.145:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56036517a1a0 con 0x560365151000 2024-03-24T11:40:49.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:49 smithi145 bash[60221]: 387fa4d011fea73b0e1e11d44a57022473cea2a2bd2cd6ee257e60b5cc0b372d 2024-03-24T11:40:49.362 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:49 smithi145 systemd[1]: Started Ceph mgr.y for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:40:50.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:49.726+0000 7f5ceac6e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:40:50.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:49.831+0000 7f5ceac6e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:40:50.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.541 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.541 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.541 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.541 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.541 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.541 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.541 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:50.539+0000 7f5ceac6e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:40:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:50 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:50.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:50.634+0000 7f5ceac6e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:40:50.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:50.780+0000 7f5ceac6e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:40:51.176 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:50.885+0000 7f5ceac6e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:40:51.176 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:50.974+0000 7f5ceac6e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:40:51.482 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:51 smithi145 ceph-mon[29394]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:51.483 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:51.274+0000 7f5ceac6e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:40:51.483 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:51.399+0000 7f5ceac6e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:40:51.483 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:51 smithi145 ceph-mon[24853]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:51.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:51 smithi149 ceph-mon[25685]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:51.811 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:51.494+0000 7f5ceac6e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:40:51.811 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:51.602+0000 7f5ceac6e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:40:52.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:51.810+0000 7f5ceac6e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:40:53.229 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:52 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:52.865+0000 7f5ceac6e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:40:53.229 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:53.006+0000 7f5ceac6e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:40:53.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-mon[29394]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:53.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-mon[29394]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:53.612 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:53.362+0000 7f5ceac6e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:40:53.612 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:53.458+0000 7f5ceac6e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:40:53.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-mon[24853]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:53.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-mon[24853]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:53 smithi149 ceph-mon[25685]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:53 smithi149 ceph-mon[25685]: from='client.25114 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:54.031 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:53.663+0000 7f5ceac6e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:40:54.031 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:53.869+0000 7f5ceac6e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:40:54.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:54.169+0000 7f5ceac6e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:40:54.694 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:54.437+0000 7f5ceac6e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:40:54.695 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:54.694+0000 7f5ceac6e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:40:55.576 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[29394]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:55.577 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:55.577 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:55.577 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:55.577 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[24853]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:55.577 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:55.577 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:55.577 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:55 smithi149 ceph-mon[25685]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:55 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:55.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:55 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:55.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:55.566+0000 7f5ceac6e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:40:56.446 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:56.194+0000 7f5ceac6e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:40:56.446 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:56.302+0000 7f5ceac6e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:40:56.701 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:56.536+0000 7f5ceac6e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:40:56.701 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:40:56] ENGINE Bus STARTING 2024-03-24T11:40:56.702 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: CherryPy Checker: 2024-03-24T11:40:56.702 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: The Application mounted at '' has an empty config. 2024-03-24T11:40:56.702 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:40:56.702 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:40:56] ENGINE Serving on http://:::9283 2024-03-24T11:40:56.702 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:40:56] ENGINE Bus STARTED 2024-03-24T11:40:56.966 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:56.966 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:56.966 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: Standby manager daemon y restarted 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: Standby manager daemon y started 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: Standby manager daemon y restarted 2024-03-24T11:40:56.967 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: Standby manager daemon y started 2024-03-24T11:40:56.968 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T11:40:56.968 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:40:56.968 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T11:40:56.968 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:56 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:40:56.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:40:56] "GET /metrics HTTP/1.1" 200 38158 "" "Prometheus/2.33.4" 2024-03-24T11:40:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:40:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: Standby manager daemon y restarted 2024-03-24T11:40:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: Standby manager daemon y started 2024-03-24T11:40:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T11:40:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:40:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T11:40:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:56 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:40:57.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:57 smithi145 ceph-mon[24853]: mgrmap e26: x(active, since 77s), standbys: y 2024-03-24T11:40:57.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:57 smithi145 ceph-mon[29394]: mgrmap e26: x(active, since 77s), standbys: y 2024-03-24T11:40:57.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:57 smithi149 ceph-mon[25685]: mgrmap e26: x(active, since 77s), standbys: y 2024-03-24T11:40:58.935 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:40:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:40:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:40:59.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:40:58 smithi145 ceph-mon[24853]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:59.041 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:40:58 smithi145 ceph-mon[29394]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:59.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:40:58 smithi149 ceph-mon[25685]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:00.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:41:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:41:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:00 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:00.374 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:41:00.375 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:00 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:01.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:01 smithi145 ceph-mon[24853]: Reconfiguring iscsi.foo.smithi145.hnqhcg (dependencies changed)... 2024-03-24T11:41:01.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:01 smithi145 ceph-mon[24853]: Reconfiguring daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:41:01.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:01 smithi145 ceph-mon[24853]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.342 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:01 smithi145 ceph-mon[29394]: Reconfiguring iscsi.foo.smithi145.hnqhcg (dependencies changed)... 2024-03-24T11:41:01.342 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:01 smithi145 ceph-mon[29394]: Reconfiguring daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:41:01.342 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:01 smithi145 ceph-mon[29394]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:01 smithi149 ceph-mon[25685]: Reconfiguring iscsi.foo.smithi145.hnqhcg (dependencies changed)... 2024-03-24T11:41:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:01 smithi149 ceph-mon[25685]: Reconfiguring daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:41:01.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:01 smithi149 ceph-mon[25685]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:03.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-mon[25685]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:03.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:03.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-mon[25685]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-24T11:41:03.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-24T11:41:03.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-mon[25685]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:41:03.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-mon[25685]: Standby manager daemon x started 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:02.928+0000 7feed119d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Creating ceph-iscsi config... 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Traceback (most recent call last): 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return _run_code(code, main_globals, None, 2024-03-24T11:41:03.226 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: exec(code, run_globals) 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:03.227 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Traceback (most recent call last): 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return future.result(timeout) 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return self.__get_result() 2024-03-24T11:41:03.228 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: raise self._exception 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: config_blobs=daemon_spec.final_config, 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Creating ceph-iscsi config... 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:03.229 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: Traceback (most recent call last): 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: return _run_code(code, main_globals, None, 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: exec(code, run_globals) 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:03.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:03.231 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:03.231 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:03.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[24853]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:03.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:03.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[24853]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-24T11:41:03.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-24T11:41:03.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[24853]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:41:03.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[24853]: Standby manager daemon x started 2024-03-24T11:41:03.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[29394]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:03.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:03.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[29394]: from='mgr.24851 172.21.15.149:0/2181429939' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-24T11:41:03.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-24T11:41:03.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[29394]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:41:03.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-mon[29394]: Standby manager daemon x started 2024-03-24T11:41:03.416 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:41:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:41:04.227 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: return future.result(timeout) 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.__get_result() 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: raise self._exception 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: config_blobs=daemon_spec.final_config, 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:04.228 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:04.229 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:04.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:03.943+0000 7fef3eae9700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-24T11:41:04.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ignoring --setuser ceph since I am not root 2024-03-24T11:41:04.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ignoring --setgroup ceph since I am not root 2024-03-24T11:41:04.230 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:03 smithi149 ceph-mgr[46949]: -- 172.21.15.149:0/3960241028 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c2b14941a0 con 0x55c2b146a800 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: Failing over to other MGR 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.24851 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-24T11:41:04.231 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: mgrmap e27: y(active, starting, since 0.999431s), standbys: x 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:41:04.232 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:41:04.233 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:41:04.233 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:41:04.233 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:41:04.233 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:41:04.233 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: Manager daemon y is now available 2024-03-24T11:41:04.233 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:04.233 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:04.234 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:04.234 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:04.234 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:04.234 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:04.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:41:04.362 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:41:04.363 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: return future.result(timeout) 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: return self.__get_result() 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: raise self._exception 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: config_blobs=daemon_spec.final_config, 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:41:04.364 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:04.365 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:41:04.366 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-24T11:41:04.367 INFO:journalctl@ceph.mon.a.smithi145.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: return future.result(timeout) 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: return self.__get_result() 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: raise self._exception 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: config_blobs=daemon_spec.final_config, 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:41:04.368 INFO:journalctl@ceph.mon.a.smithi145.stdout: Creating ceph-iscsi config... 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: Non-zero exit code 1 from systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: Traceback (most recent call last): 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:41:04.369 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:41:04.370 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:03] ENGINE Bus STOPPING 2024-03-24T11:41:04.370 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:41:04.370 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:03] ENGINE Bus STOPPED 2024-03-24T11:41:04.370 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:04] ENGINE Bus STARTING 2024-03-24T11:41:04.370 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:04] ENGINE Serving on http://:::9283 2024-03-24T11:41:04.370 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:04] ENGINE Bus STARTED 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: Failing over to other MGR 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.24851 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: mgrmap e27: y(active, starting, since 0.999431s), standbys: x 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:41:04.371 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:41:04.372 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1065, in deploy_daemon 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout: RuntimeError: Failed command: systemctl restart ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: Failing over to other MGR 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.24851 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: mgrmap e27: y(active, starting, since 0.999431s), standbys: x 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:41:04.373 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:41:04.374 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: Manager daemon y is now available 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:41:04.375 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: Manager daemon y is now available 2024-03-24T11:41:04.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:04.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:04.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:04.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:04.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:04.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:04 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:04.967 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:41:04.967 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (58s) 10s ago 5m 16.3M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:41:04.967 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (21s) 5s ago 5m 42.9M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:41:04.967 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (43s) 10s ago 4m 46.0M - 3.5 e1d6a67b021e 58f5a9fadacf 2024-03-24T11:41:04.967 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283 running (101s) 5s ago 10m 504M - 19.0.0-1667-gdb0330b1 4f325f8440e9 25419047a080 2024-03-24T11:41:04.967 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (16s) 10s ago 12m 91.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (12m) 10s ago 12m 54.3M 2048M 17.2.0 e1d6a67b021e ca31a24577e6 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (10m) 5s ago 10m 44.4M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (10m) 10s ago 10m 41.2M 2048M 17.2.0 e1d6a67b021e 0b79dbdf9a4c 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (67s) 10s ago 6m 11.6M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (37s) 5s ago 6m 9625k - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (10m) 10s ago 10m 51.7M 2736M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (9m) 10s ago 9m 53.5M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (9m) 10s ago 9m 48.7M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (8m) 10s ago 8m 50.7M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (8m) 5s ago 8m 54.4M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (8m) 5s ago 8m 49.2M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (7m) 5s ago 7m 48.2M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:41:04.968 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (7m) 5s ago 7m 51.2M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:41:04.969 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (39s) 5s ago 6m 51.3M - 2.33.4 514e6a882f6e 5a4dec45ed4b 2024-03-24T11:41:04.969 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (5m) 10s ago 5m 69.5M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:41:04.969 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (5m) 5s ago 5m 73.4M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:41:04.969 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (4m) 10s ago 4m 70.7M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:41:04.969 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (4m) 5s ago 4m 72.1M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:41:04.974 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:04.530+0000 7f8906213200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:41:04.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:04.628+0000 7f8906213200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": {}, 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:41:05.379 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:41:05.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:05.325+0000 7f8906213200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:41:05.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:05.423+0000 7f8906213200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:41:05.686 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:05.571+0000 7f8906213200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:41:05.686 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:05.685+0000 7f8906213200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:41:05.719 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:41:05.719 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:41:05.719 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:41:05.719 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:41:05.719 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:41:05.719 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr" 2024-03-24T11:41:05.719 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:41:05.720 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T11:41:05.720 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "", 2024-03-24T11:41:05.720 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:41:05.720 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:41:05.944 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:05.770+0000 7f8906213200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:41:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:05 smithi145 ceph-mon[24853]: mgrmap e28: y(active, since 2s), standbys: x 2024-03-24T11:41:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:05 smithi145 ceph-mon[24853]: [24/Mar/2024:11:41:05] ENGINE Bus STARTING 2024-03-24T11:41:06.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:05 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/3318763385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:05 smithi145 ceph-mon[29394]: mgrmap e28: y(active, since 2s), standbys: x 2024-03-24T11:41:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:05 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:05] ENGINE Bus STARTING 2024-03-24T11:41:06.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:05 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/3318763385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:06.128 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:41:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-mon[25685]: mgrmap e28: y(active, since 2s), standbys: x 2024-03-24T11:41:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:05] ENGINE Bus STARTING 2024-03-24T11:41:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:05 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/3318763385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:06.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:06.056+0000 7f8906213200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:41:06.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:06.171+0000 7f8906213200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:06.578 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:06.258+0000 7f8906213200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:41:06.578 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:06.358+0000 7f8906213200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:41:06.578 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:06.577+0000 7f8906213200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:41:07.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:05] ENGINE Serving on http://172.21.15.145:8765 2024-03-24T11:41:07.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:05] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:41:07.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:05] ENGINE Bus STARTED 2024-03-24T11:41:07.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-mon[25685]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:07.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-mon[25685]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:07.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:06 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2969632525' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:41:07.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:05] ENGINE Serving on http://172.21.15.145:8765 2024-03-24T11:41:07.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:05] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:05] ENGINE Bus STARTED 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[29394]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[29394]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/2969632525' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[24853]: [24/Mar/2024:11:41:05] ENGINE Serving on http://172.21.15.145:8765 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[24853]: [24/Mar/2024:11:41:05] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[24853]: [24/Mar/2024:11:41:05] ENGINE Bus STARTED 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[24853]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[24853]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:07.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:06 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/2969632525' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:41:07.955 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:07.597+0000 7f8906213200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:41:07.955 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:07.732+0000 7f8906213200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:41:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:07 smithi149 ceph-mon[25685]: mgrmap e29: y(active, since 4s), standbys: x 2024-03-24T11:41:08.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:08.095+0000 7f8906213200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:41:08.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:08.187+0000 7f8906213200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:41:08.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:07 smithi145 ceph-mon[29394]: mgrmap e29: y(active, since 4s), standbys: x 2024-03-24T11:41:08.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:07 smithi145 ceph-mon[24853]: mgrmap e29: y(active, since 4s), standbys: x 2024-03-24T11:41:08.589 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:08.396+0000 7f8906213200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:08.901 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:08.588+0000 7f8906213200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:41:09.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-mon[25685]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:09.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.190 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:08.900+0000 7f8906213200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:09.190 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:09.188+0000 7f8906213200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:09.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[24853]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:09.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[29394]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:08 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:09.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:09.448+0000 7f8906213200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[24853]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:41:10.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:10 smithi145 ceph-mon[29394]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:41:10.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:10.311+0000 7f8906213200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:41:10.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-24T11:41:10.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:10.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:41:10.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-mon[25685]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:41:11.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:10.907+0000 7f8906213200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:41:11.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:11.008+0000 7f8906213200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:41:11.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[29394]: Standby manager daemon x restarted 2024-03-24T11:41:11.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[29394]: Standby manager daemon x started 2024-03-24T11:41:11.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:11.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[24853]: Standby manager daemon x restarted 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[24853]: Standby manager daemon x started 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:11.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:11 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-mon[25685]: Standby manager daemon x restarted 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-mon[25685]: Standby manager daemon x started 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/1856855538' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:11.229+0000 7f8906213200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:41:11.727 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:41:11] ENGINE Bus STARTING 2024-03-24T11:41:11.728 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: CherryPy Checker: 2024-03-24T11:41:11.728 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: The Application mounted at '' has an empty config. 2024-03-24T11:41:11.728 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: 2024-03-24T11:41:11.728 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:41:11] ENGINE Serving on http://:::9283 2024-03-24T11:41:11.728 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: [24/Mar/2024:11:41:11] ENGINE Bus STARTED 2024-03-24T11:41:12.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:12 smithi149 ceph-mon[25685]: mgrmap e30: y(active, since 8s), standbys: x 2024-03-24T11:41:12.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:12 smithi149 ceph-mon[25685]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:41:12.745 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:12 smithi145 ceph-mon[24853]: mgrmap e30: y(active, since 8s), standbys: x 2024-03-24T11:41:12.745 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:12 smithi145 ceph-mon[24853]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:41:12.746 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:12 smithi145 ceph-mon[29394]: mgrmap e30: y(active, since 8s), standbys: x 2024-03-24T11:41:12.746 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:12 smithi145 ceph-mon[29394]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[29394]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:13.911 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:13.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:13.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:13.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:41:13.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:41:13.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:41:13.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:13 smithi145 ceph-mon[24853]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:41:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:13 smithi149 ceph-mon[25685]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:41:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:41:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:41:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:41:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:14 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/4115798248' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/4115798248' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:41:15.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:15.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:41:15.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: Reconfiguring daemon prometheus.a on smithi149 2024-03-24T11:41:15.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:14 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/4115798248' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:41:16.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:15 smithi145 ceph-mon[24853]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:41:16.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:15 smithi145 ceph-mon[29394]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:41:16.141 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:15 smithi149 ceph-mon[25685]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:41:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:16 smithi145 ceph-mon[29394]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:41:17.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:16 smithi145 ceph-mon[24853]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:41:17.224 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:16 smithi149 ceph-mon[25685]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:41:17.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[46945]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:41:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-24T11:41:17.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 systemd[1]: Stopping Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:41:17.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 bash[53874]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-24T11:41:17.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.528Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:41:17.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.528Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:41:17.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.528Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:41:17.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.528Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.528Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.528Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.529Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.529Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.529Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.529Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.529Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[50565]: ts=2024-03-24T11:41:17.529Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-24T11:41:17.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:17 smithi149 podman[53887]: 2024-03-24 11:41:17.539125186 +0000 UTC m=+0.128366067 container died 5a4dec45ed4b5659e912831994a1026d31cded5c2ee0bb862767f74bf92b0dfd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:18.388 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 podman[53887]: 2024-03-24 11:41:18.137481217 +0000 UTC m=+0.726722152 container cleanup 5a4dec45ed4b5659e912831994a1026d31cded5c2ee0bb862767f74bf92b0dfd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:18.388 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 bash[53887]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:41:18.388 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 bash[53916]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-24T11:41:18.647 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 podman[53901]: 2024-03-24 11:41:18.388527875 +0000 UTC m=+0.851939580 container remove 5a4dec45ed4b5659e912831994a1026d31cded5c2ee0bb862767f74bf92b0dfd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:18.648 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T11:41:18.648 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 systemd[1]: Stopped Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:41:18.648 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 systemd[1]: Starting Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:41:18.648 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 podman[53982]: 2024-03-24 11:41:18.523807148 +0000 UTC m=+0.019824520 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-24T11:41:18.648 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 podman[53982]: 2024-03-24 11:41:18.648103148 +0000 UTC m=+0.144120506 container create 595718fc2704a4f5f9efa27867c67661c07e9e207f94d16242dc11a6bff796c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:19.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-mon[25685]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:41:19.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 podman[53982]: 2024-03-24 11:41:18.901512336 +0000 UTC m=+0.397529703 container init 595718fc2704a4f5f9efa27867c67661c07e9e207f94d16242dc11a6bff796c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:19.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 podman[53982]: 2024-03-24 11:41:18.904437401 +0000 UTC m=+0.400454766 container start 595718fc2704a4f5f9efa27867c67661c07e9e207f94d16242dc11a6bff796c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:41:19.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.927Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-24T11:41:19.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.927Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-24T11:41:19.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.927Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-24T11:41:19.225 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.927Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi149 (none))" 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.927Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.927Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.929Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.929Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.930Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.932Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.932Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.779µs 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:18.932Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 bash[53982]: 595718fc2704a4f5f9efa27867c67661c07e9e207f94d16242dc11a6bff796c8 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:18 smithi149 systemd[1]: Started Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:41:19.226 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.126Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-24T11:41:19.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:18 smithi145 ceph-mon[29394]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:41:19.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:18 smithi145 ceph-mon[24853]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:41:19.685 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.393Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-24T11:41:19.685 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.447Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-24T11:41:19.685 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.447Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-24T11:41:19.685 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.447Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.981µs wal_replay_duration=515.368208ms total_replay_duration=515.422628ms 2024-03-24T11:41:19.685 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.451Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:41:19.685 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.451Z caller=main.go:947 level=info msg="TSDB started" 2024-03-24T11:41:19.686 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.451Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:41:19.686 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.471Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.040672ms db_storage=718ns remote_storage=1.597µs web_handler=337ns query_engine=1.092µs scrape=379.765µs scrape_sd=91.1µs notify=10.476µs notify_sd=8.876µs rules=19.29989ms 2024-03-24T11:41:19.686 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:19.471Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-24T11:41:20.220 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:20 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.220 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:20 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.220 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:20 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:20.220 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:20 smithi149 ceph-mon[25685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:20.220 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:20 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.220 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:20 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:20.220 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:20 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[24853]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[29394]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:20.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:20.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:20 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:21.334 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:21 smithi149 ceph-mon[25685]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:41:21.334 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:21 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:21.334 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:21 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:21.334 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:21 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:41:21.334 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:21 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:21.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[24853]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:41:21.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:21.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:21.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:41:21.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[29394]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:41:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:41:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:41:21.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:21 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:22.337 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:22 smithi149 ceph-mon[25685]: Upgrade: Updating mgr.x 2024-03-24T11:41:22.338 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:22 smithi149 ceph-mon[25685]: Deploying daemon mgr.x on smithi149 2024-03-24T11:41:22.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:22 smithi145 ceph-mon[24853]: Upgrade: Updating mgr.x 2024-03-24T11:41:22.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:22 smithi145 ceph-mon[24853]: Deploying daemon mgr.x on smithi149 2024-03-24T11:41:22.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:22 smithi145 ceph-mon[29394]: Upgrade: Updating mgr.x 2024-03-24T11:41:22.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:22 smithi145 ceph-mon[29394]: Deploying daemon mgr.x on smithi149 2024-03-24T11:41:22.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:22 smithi149 systemd[1]: Stopping Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:41:23.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:23 smithi149 ceph-mon[25685]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:41:23.189 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:22 smithi149 podman[54335]: 2024-03-24 11:41:22.885195363 +0000 UTC m=+0.167501320 container died 25419047a080aab568683f6dac6fd20669e83a3b5d5c621ee22e68d24560b2a8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:41:23.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:23 smithi145 ceph-mon[24853]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:41:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:23 smithi145 ceph-mon[29394]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:41:23.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 podman[54335]: 2024-03-24 11:41:23.287798316 +0000 UTC m=+0.570104312 container cleanup 25419047a080aab568683f6dac6fd20669e83a3b5d5c621ee22e68d24560b2a8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:41:23.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 bash[54335]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x 2024-03-24T11:41:23.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 podman[54349]: 2024-03-24 11:41:23.505459473 +0000 UTC m=+0.618748258 container remove 25419047a080aab568683f6dac6fd20669e83a3b5d5c621ee22e68d24560b2a8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T11:41:23.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:41:23.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-24T11:41:23.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 systemd[1]: Stopped Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:41:23.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service: Consumed 26.585s CPU time. 2024-03-24T11:41:23.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:23 smithi149 systemd[1]: Starting Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:41:24.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 podman[54480]: 2024-03-24 11:41:24.042477766 +0000 UTC m=+0.019222605 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:41:24.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 podman[54480]: 2024-03-24 11:41:24.165221457 +0000 UTC m=+0.141966277 container create bba2809eb6bea83a904974bad2f3e70b231771b056648e0ba44fcbf23e3aea87 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:41:24.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 podman[54480]: 2024-03-24 11:41:24.367716381 +0000 UTC m=+0.344461217 container init bba2809eb6bea83a904974bad2f3e70b231771b056648e0ba44fcbf23e3aea87 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T11:41:24.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 podman[54480]: 2024-03-24 11:41:24.370329587 +0000 UTC m=+0.347074411 container start bba2809eb6bea83a904974bad2f3e70b231771b056648e0ba44fcbf23e3aea87 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307) 2024-03-24T11:41:24.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 ceph-mgr[54499]: -- 172.21.15.149:0/230200834 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5641e3a601a0 con 0x5641e3a36800 2024-03-24T11:41:24.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 bash[54480]: bba2809eb6bea83a904974bad2f3e70b231771b056648e0ba44fcbf23e3aea87 2024-03-24T11:41:24.725 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 systemd[1]: Started Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:41:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:25 smithi149 ceph-mon[25685]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:41:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:25 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:25 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:25.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:24 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:24.947+0000 7fe4c4f68200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:41:25.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:25.047+0000 7fe4c4f68200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:41:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:25 smithi145 ceph-mon[24853]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:41:25.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:25 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:25 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:25 smithi145 ceph-mon[29394]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:41:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:25 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:25.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:25 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:25.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:41:25] "GET /metrics HTTP/1.1" 200 38246 "" "Prometheus/2.33.4" 2024-03-24T11:41:25.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:25.740+0000 7fe4c4f68200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:41:25.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:25.832+0000 7fe4c4f68200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:41:26.281 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:26 smithi149 ceph-mon[25685]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:26.281 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:25.994+0000 7fe4c4f68200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:41:26.281 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:26 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:26.094+0000 7fe4c4f68200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:41:26.281 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:26 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:26.179+0000 7fe4c4f68200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:41:26.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:26 smithi145 ceph-mon[24853]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:26.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:26 smithi145 ceph-mon[29394]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:26.573 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:26 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:26.457+0000 7fe4c4f68200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:41:26.573 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:26 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:26.572+0000 7fe4c4f68200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:26.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:26 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:26.685+0000 7fe4c4f68200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:41:26.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:26 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:26.814+0000 7fe4c4f68200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:41:27.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:27 smithi145 ceph-mon[24853]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-24T11:41:27.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:27 smithi145 ceph-mon[29394]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-24T11:41:27.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:27 smithi149 ceph-mon[25685]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-24T11:41:27.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:27.006+0000 7fe4c4f68200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:41:28.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:28.027+0000 7fe4c4f68200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:41:28.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:28.159+0000 7fe4c4f68200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:41:28.776 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:41:28.776 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:28.520+0000 7fe4c4f68200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:41:28.776 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:28.611+0000 7fe4c4f68200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:41:29.072 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:28.822+0000 7fe4c4f68200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:29.073 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:29 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:29.013+0000 7fe4c4f68200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:41:29.329 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:29 smithi149 ceph-mon[25685]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:29.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:29 smithi145 ceph-mon[24853]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:29.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:29 smithi145 ceph-mon[29394]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:29.611 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:29 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:29.328+0000 7fe4c4f68200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:29.866 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:29 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:29.610+0000 7fe4c4f68200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:29.866 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:29 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:29.865+0000 7fe4c4f68200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:41:30.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:30 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:30.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:30 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:30.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:30 smithi145 ceph-mon[24853]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:30.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:30 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:30.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:30 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:30.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:30 smithi145 ceph-mon[29394]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:30.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:30 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:30.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:30 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:30.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:30 smithi149 ceph-mon[25685]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:31.024 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:30 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:30.753+0000 7fe4c4f68200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:41:31.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:31.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:31.626 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:31.383+0000 7fe4c4f68200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:41:31.626 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:31.490+0000 7fe4c4f68200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:41:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:31 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:31.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:31 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:31 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:31.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:31 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:31.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:31.729+0000 7fe4c4f68200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:41:31.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: [24/Mar/2024:11:41:31] ENGINE Bus STARTING 2024-03-24T11:41:31.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: CherryPy Checker: 2024-03-24T11:41:31.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: The Application mounted at '' has an empty config. 2024-03-24T11:41:31.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:31.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: [24/Mar/2024:11:41:31] ENGINE Serving on http://:::9283 2024-03-24T11:41:31.976 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: [24/Mar/2024:11:41:31] ENGINE Bus STARTED 2024-03-24T11:41:32.592 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:32 smithi149 ceph-mon[25685]: Standby manager daemon x restarted 2024-03-24T11:41:32.592 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:32 smithi149 ceph-mon[25685]: Standby manager daemon x started 2024-03-24T11:41:32.592 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:32 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:32.592 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:32 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:32.592 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:32 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:32.592 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:32 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:32.592 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:32 smithi149 ceph-mon[25685]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:32.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[24853]: Standby manager daemon x restarted 2024-03-24T11:41:32.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[24853]: Standby manager daemon x started 2024-03-24T11:41:32.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[24853]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[24853]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[29394]: Standby manager daemon x restarted 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[29394]: Standby manager daemon x started 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/3266894227' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:32 smithi145 ceph-mon[29394]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:33.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:33 smithi149 ceph-mon[25685]: mgrmap e31: y(active, since 29s), standbys: x 2024-03-24T11:41:33.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:33 smithi145 ceph-mon[24853]: mgrmap e31: y(active, since 29s), standbys: x 2024-03-24T11:41:33.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:33 smithi145 ceph-mon[29394]: mgrmap e31: y(active, since 29s), standbys: x 2024-03-24T11:41:35.048 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: Upgrade: Setting container_image for all mgr 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-24T11:41:35.049 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:35.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: Upgrade: Setting container_image for all mgr 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-24T11:41:35.051 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:41:35.052 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-24T11:41:35.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:35.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: Upgrade: Setting container_image for all mgr 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-24T11:41:35.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-24T11:41:35.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-24T11:41:35.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-24T11:41:35.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:41:35.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:35 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-24T11:41:35.840 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:41:35] "GET /metrics HTTP/1.1" 200 38247 "" "Prometheus/2.33.4" 2024-03-24T11:41:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[24853]: Upgrade: It appears safe to stop mon.a 2024-03-24T11:41:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[24853]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:36.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[24853]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[24853]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[29394]: Upgrade: It appears safe to stop mon.a 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[29394]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:41:36.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:36 smithi145 ceph-mon[29394]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:36.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:36 smithi149 ceph-mon[25685]: Upgrade: It appears safe to stop mon.a 2024-03-24T11:41:36.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:36 smithi149 ceph-mon[25685]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:36.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:36 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='mgr.y' 2024-03-24T11:41:36.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:36 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:41:36.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:36 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:41:36.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:36 smithi149 ceph-mon[25685]: from='mgr.15300 172.21.15.145:0/1100357661' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:36.489 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:41:37.143 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:41:37.143 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (90s) 28s ago 6m 16.4M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (53s) 7s ago 6m 45.2M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 unknown 28s ago 5m - - 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (13s) 7s ago 10m 66.3M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (48s) 28s ago 12m 474M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (12m) 28s ago 12m 55.5M 2048M 17.2.0 e1d6a67b021e ca31a24577e6 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (11m) 7s ago 11m 47.9M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (11m) 28s ago 11m 42.9M 2048M 17.2.0 e1d6a67b021e 0b79dbdf9a4c 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (99s) 28s ago 7m 11.2M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (70s) 7s ago 7m 16.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (10m) 28s ago 10m 51.9M 2736M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (10m) 28s ago 10m 53.8M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (9m) 28s ago 9m 49.0M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (9m) 28s ago 9m 51.1M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:41:37.144 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (9m) 7s ago 9m 55.0M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (8m) 7s ago 8m 49.7M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (8m) 7s ago 8m 48.9M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (7m) 7s ago 7m 51.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (18s) 7s ago 7m 52.0M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (5m) 28s ago 5m 70.1M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (5m) 7s ago 5m 74.2M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (5m) 28s ago 5m 71.2M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:41:37.145 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (5m) 7s ago 5m 73.3M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:41:37.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:37 smithi145 ceph-mon[24853]: Upgrade: Updating mon.a 2024-03-24T11:41:37.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:37 smithi145 ceph-mon[24853]: Deploying daemon mon.a on smithi145 2024-03-24T11:41:37.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:37 smithi145 ceph-mon[24853]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:37.332 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:37 smithi145 ceph-mon[29394]: Upgrade: Updating mon.a 2024-03-24T11:41:37.332 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:37 smithi145 ceph-mon[29394]: Deploying daemon mon.a on smithi145 2024-03-24T11:41:37.332 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:37 smithi145 ceph-mon[29394]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:37.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:37 smithi149 ceph-mon[25685]: Upgrade: Updating mon.a 2024-03-24T11:41:37.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:37 smithi149 ceph-mon[25685]: Deploying daemon mon.a on smithi145 2024-03-24T11:41:37.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:37 smithi149 ceph-mon[25685]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:41:37.540 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": {}, 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:41:37.541 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr" 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "2/23 daemons upgraded", 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading mon daemons", 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:41:37.957 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:41:38.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:37 smithi145 systemd[1]: Stopping Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[29394]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[29394]: from='client.25309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[29394]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1266097742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[24853]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[24853]: from='client.25309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[24853]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-mon[24853]: from='client.? 172.21.15.145:0/1266097742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:38.332 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a[24849]: 2024-03-24T11:41:38.051+0000 7f50bd380700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:41:38.333 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a[24849]: 2024-03-24T11:41:38.051+0000 7f50bd380700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T11:41:38.333 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 podman[64472]: 2024-03-24 11:41:38.072262516 +0000 UTC m=+0.134661856 container died ca31a24577e69c1ac2ba3d38b4326a941d78f341dd9b95e6b414df2973548680 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, distribution-scope=public, version=8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD) 2024-03-24T11:41:38.355 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-24T11:41:38.356 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:41:38.356 INFO:teuthology.orchestra.run.smithi145.stdout: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:41:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:38 smithi149 ceph-mon[25685]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:38 smithi149 ceph-mon[25685]: from='client.25309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:38 smithi149 ceph-mon[25685]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:38.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:38 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1266097742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:41:38.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 podman[64472]: 2024-03-24 11:41:38.439083088 +0000 UTC m=+0.501482443 container cleanup ca31a24577e69c1ac2ba3d38b4326a941d78f341dd9b95e6b414df2973548680 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., release=754, vcs-type=git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8) 2024-03-24T11:41:38.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 bash[64472]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:41:38.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:38 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:41:39.054 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 podman[64509]: 2024-03-24 11:41:38.723572831 +0000 UTC m=+0.648881664 container remove ca31a24577e69c1ac2ba3d38b4326a941d78f341dd9b95e6b414df2973548680 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, release=754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_CLEAN=True, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, name=centos-stream) 2024-03-24T11:41:39.054 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T11:41:39.054 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 systemd[1]: Stopped Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:41:39.054 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:38 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service: Consumed 13.760s CPU time. 2024-03-24T11:41:39.054 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 systemd[1]: Starting Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:41:39.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 podman[64643]: 2024-03-24 11:41:39.174059633 +0000 UTC m=+0.018594595 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:41:39.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 podman[64643]: 2024-03-24 11:41:39.299626017 +0000 UTC m=+0.144161011 container create 7ad6536c6cf9d5bb70bf6831973cbd38966d2840abfc03f66b0974f85a43f11a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T11:41:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 podman[64643]: 2024-03-24 11:41:39.502305597 +0000 UTC m=+0.346840522 container init 7ad6536c6cf9d5bb70bf6831973cbd38966d2840abfc03f66b0974f85a43f11a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T11:41:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 podman[64643]: 2024-03-24 11:41:39.505381755 +0000 UTC m=+0.349916682 container start 7ad6536c6cf9d5bb70bf6831973cbd38966d2840abfc03f66b0974f85a43f11a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T11:41:39.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:41:39.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T11:41:39.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: pidfile_write: ignore empty --pid-file 2024-03-24T11:41:39.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: load: jerasure load: lrc 2024-03-24T11:41:39.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:41:39.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Git sha 0 2024-03-24T11:41:39.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:41:39.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: DB SUMMARY 2024-03-24T11:41:39.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: DB Session ID: HDZIXEA3J7OPRLB1AVO9 2024-03-24T11:41:39.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:41:39.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:41:39.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-03-24T11:41:39.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-03-24T11:41:39.865 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3806703 ; 2024-03-24T11:41:39.865 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:41:39.865 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:41:39.865 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:41:39.865 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:41:39.865 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:41:39.866 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:41:39.866 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.env: 0x55e2d10fbae0 2024-03-24T11:41:39.866 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:41:39.866 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.info_log: 0x55e2d2fb98c0 2024-03-24T11:41:39.866 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:41:39.866 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.statistics: (nil) 2024-03-24T11:41:39.869 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.use_fsync: 0 2024-03-24T11:41:39.869 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:41:39.869 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:41:39.869 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:41:39.869 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.db_log_dir: 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.wal_dir: 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:41:39.870 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.write_buffer_manager: 0x55e2d2f8cdc0 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:41:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.unordered_write: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.row_cache: None 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.wal_filter: None 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.wal_compression: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:41:39.872 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:41:39.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_open_files: -1 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Compression algorithms supported: 2024-03-24T11:41:39.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kZSTD supported: 0 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.merge_operator: 2024-03-24T11:41:39.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_filter: None 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e2d2fb99c0) 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_type: 0 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout: data_block_index_type: 0 2024-03-24T11:41:39.876 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_shortening: 1 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: checksum: 4 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: no_block_cache: 0 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache: 0x55e2d2fbd090 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_options: 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: capacity : 536870912 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: num_shard_bits : 4 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: strict_capacity_limit : 0 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_compressed: (nil) 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: persistent_cache: (nil) 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_size: 4096 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_size_deviation: 10 2024-03-24T11:41:39.877 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_restart_interval: 16 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_block_restart_interval: 1 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: metadata_block_size: 4096 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: partition_filters: 0 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: use_delta_encoding: 1 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: filter_policy: bloomfilter 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: whole_key_filtering: 1 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: verify_compression: 0 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: format_version: 5 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: enable_index_compression: 1 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_align: 0 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: max_auto_readahead_size: 262144 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: prepopulate_block_cache: 0 2024-03-24T11:41:39.878 INFO:journalctl@ceph.mon.a.smithi145.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression: NoCompression 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.num_levels: 7 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:41:39.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:41:39.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:41:39.881 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:41:39.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:41:39.883 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.ttl: 2592000 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:41:39.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12082, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d2c68081-9ae5-40d1-9e17-1c2876f6a43d 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280499534704, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280499544606, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3511527, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12083, "largest_seqno": 13655, "table_properties": {"data_size": 3503970, "index_size": 4745, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1797, "raw_key_size": 16461, "raw_average_key_size": 23, "raw_value_size": 3488839, "raw_average_value_size": 5027, "num_data_blocks": 215, "num_entries": 694, "num_filter_entries": 694, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280499, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d2c68081-9ae5-40d1-9e17-1c2876f6a43d", "db_session_id": "HDZIXEA3J7OPRLB1AVO9", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280499544706, "job": 1, "event": "recovery_finished"} 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:41:39.885 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e2d30ae000 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: DB pointer 0x55e2d3098000 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** DB Stats ** 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:41:39.886 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: L0 1/0 3.35 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 742.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: L6 1/0 7.48 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: Sum 2/0 10.83 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 742.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 742.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 742.2 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:41:39.887 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative compaction: 0.00 GB write, 229.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval compaction: 0.00 GB write, 229.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: Block cache BinnedLRUCache@0x55e2d2fbd090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:41:39.888 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: starting mon.a rank 0 at public addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] at bind addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???) e3 preinit fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???).mds e1 new map 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???).mds e1 print_map 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout: e1 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout: legacy client fscid: -1 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout: No filesystems configured 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 ceph-mon[64662]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-24T11:41:39.889 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 bash[64643]: 7ad6536c6cf9d5bb70bf6831973cbd38966d2840abfc03f66b0974f85a43f11a 2024-03-24T11:41:39.890 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:39 smithi145 systemd[1]: Started Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:41:42.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/1488346956' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: mon.a calling monitor election 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: fsmap 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: mgrmap e31: y(active, since 39s), standbys: x 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:41:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:41:42.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: from='mgr.15300 ' entity='' 2024-03-24T11:41:42.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mon[25685]: mgrmap e32: y(active, since 39s), standbys: x 2024-03-24T11:41:42.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: ignoring --setuser ceph since I am not root 2024-03-24T11:41:42.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: ignoring --setgroup ceph since I am not root 2024-03-24T11:41:42.977 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:42 smithi149 ceph-mgr[54499]: -- 172.21.15.149:0/4219188925 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5590988f21a0 con 0x5590988c9000 2024-03-24T11:41:43.049 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ignoring --setuser ceph since I am not root 2024-03-24T11:41:43.049 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ignoring --setgroup ceph since I am not root 2024-03-24T11:41:43.049 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mgr[60239]: -- 172.21.15.145:0/1095574054 <== mon.1 v2:172.21.15.145:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5577e69a41a0 con 0x5577e697a800 2024-03-24T11:41:43.049 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:43.049 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.049 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: from='client.? 172.21.15.145:0/1488346956' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: mon.a calling monitor election 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: fsmap 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: mgrmap e31: y(active, since 39s), standbys: x 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: from='mgr.15300 ' entity='' 2024-03-24T11:41:43.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[29394]: mgrmap e32: y(active, since 39s), standbys: x 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1488346956' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: mon.a calling monitor election 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: fsmap 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: mgrmap e31: y(active, since 39s), standbys: x 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:41:43.051 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:41:43.052 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: from='mgr.15300 ' entity='' 2024-03-24T11:41:43.052 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:42 smithi145 ceph-mon[64662]: mgrmap e32: y(active, since 39s), standbys: x 2024-03-24T11:41:43.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:43.173+0000 7f50450c6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:41:43.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:43.266+0000 7f50450c6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:41:43.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:43.157+0000 7f549755d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:41:43.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:43.252+0000 7f549755d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:41:44.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:43.899+0000 7f549755d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:41:44.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:43.993+0000 7f549755d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:41:44.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:44.139+0000 7f549755d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:41:44.243 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:43.909+0000 7f50450c6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:41:44.243 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.003+0000 7f50450c6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:41:44.243 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.146+0000 7f50450c6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:41:44.588 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:44.236+0000 7f549755d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:41:44.588 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:44.320+0000 7f549755d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:41:44.588 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:44.587+0000 7f549755d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:41:44.590 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.242+0000 7f50450c6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:41:44.590 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.325+0000 7f50450c6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:41:44.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-mon[29394]: mgrmap e33: y(active, since 40s), standbys: x 2024-03-24T11:41:44.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-mon[64662]: mgrmap e33: y(active, since 40s), standbys: x 2024-03-24T11:41:44.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.595+0000 7f50450c6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:41:44.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.705+0000 7f50450c6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:44.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.795+0000 7f50450c6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:41:44.882 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-mon[25685]: mgrmap e33: y(active, since 40s), standbys: x 2024-03-24T11:41:44.883 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:44.698+0000 7f549755d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:44.883 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:44.786+0000 7f549755d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:41:44.883 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:44 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:44.881+0000 7f549755d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:41:45.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:45.063+0000 7f549755d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:41:45.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:44.893+0000 7f50450c6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:41:45.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:45.077+0000 7f50450c6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:41:46.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:46.051+0000 7f50450c6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:41:46.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:46.183+0000 7f50450c6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:41:46.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:46.035+0000 7f549755d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:41:46.476 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:46.168+0000 7f549755d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:41:46.799 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:46.509+0000 7f549755d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:41:46.799 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:46.601+0000 7f549755d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:41:46.800 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:46.798+0000 7f549755d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:46.811 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:46.522+0000 7f50450c6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:41:46.812 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:46.614+0000 7f50450c6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:41:46.812 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:46.811+0000 7f50450c6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:47.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:46 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:46.998+0000 7f50450c6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:41:47.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:46.988+0000 7f549755d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:41:47.548 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:47.279+0000 7f549755d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:47.556 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:47.290+0000 7f50450c6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:47.555+0000 7f50450c6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:47.795+0000 7f50450c6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:41:47.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:47.547+0000 7f549755d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:41:47.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:47 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:47.787+0000 7f549755d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:41:48.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:48 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:48.607+0000 7f50450c6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:41:48.975 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:48.596+0000 7f549755d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:41:49.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:49.184+0000 7f549755d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:41:49.475 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:49.285+0000 7f549755d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:41:49.514 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:49.185+0000 7f50450c6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:41:49.514 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:49.287+0000 7f50450c6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:41:49.514 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:49.513+0000 7f50450c6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:41:49.766 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:49.506+0000 7f549755d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:41:49.766 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: [24/Mar/2024:11:41:49] ENGINE Bus STARTING 2024-03-24T11:41:49.766 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: CherryPy Checker: 2024-03-24T11:41:49.766 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: The Application mounted at '' has an empty config. 2024-03-24T11:41:49.766 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: [24/Mar/2024:11:41:49] ENGINE Serving on http://:::9283 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x[54495]: [24/Mar/2024:11:41:49] ENGINE Bus STARTED 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b[25667]: 2024-03-24T11:41:49.529+0000 7f9c6bbe1700 -1 mon.b@2(peon).osd e106 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: Standby manager daemon x restarted 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: Standby manager daemon x started 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: Active manager daemon y restarted 2024-03-24T11:41:49.767 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: Activating manager daemon y 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: mgrmap e34: y(active, starting, since 0.00793813s), standbys: x 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:41:49.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:41:49.769 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:41:49.769 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:41:49.769 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:41:49.769 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:49 smithi149 ceph-mon[25685]: Manager daemon y is now available 2024-03-24T11:41:49.773 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:49] ENGINE Bus STARTING 2024-03-24T11:41:49.773 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c[29389]: 2024-03-24T11:41:49.531+0000 7ff9508f0700 -1 mon.c@1(peon).osd e106 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-24T11:41:49.773 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: Standby manager daemon x restarted 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: Standby manager daemon x started 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: Active manager daemon y restarted 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: Activating manager daemon y 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: mgrmap e34: y(active, starting, since 0.00793813s), standbys: x 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:41:49.774 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:41:49.775 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[29394]: Manager daemon y is now available 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: Standby manager daemon x restarted 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: Standby manager daemon x started 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.? 172.21.15.149:0/2263275098' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: Active manager daemon y restarted 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: Activating manager daemon y 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: mgrmap e34: y(active, starting, since 0.00793813s), standbys: x 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:41:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:41:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-mon[64662]: Manager daemon y is now available 2024-03-24T11:41:50.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: CherryPy Checker: 2024-03-24T11:41:50.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: The Application mounted at '' has an empty config. 2024-03-24T11:41:50.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:50.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:49] ENGINE Serving on http://:::9283 2024-03-24T11:41:50.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:49 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:41:49] ENGINE Bus STARTED 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:41:50.537+0000 7f4ffb2c5700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[64662]: mgrmap e35: y(active, since 1.01048s), standbys: x 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.854 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:50.855 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.855 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.855 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:50 smithi145 ceph-mon[29394]: mgrmap e35: y(active, since 1.01048s), standbys: x 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:50.875 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:50 smithi149 ceph-mon[25685]: mgrmap e35: y(active, since 1.01048s), standbys: x 2024-03-24T11:41:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: failed to encode map e106 with expected crc 2024-03-24T11:41:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:49] ENGINE Bus STARTING 2024-03-24T11:41:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:50] ENGINE Serving on http://172.21.15.145:8765 2024-03-24T11:41:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:50] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:41:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: [24/Mar/2024:11:41:50] ENGINE Bus STARTED 2024-03-24T11:41:51.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:51.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:51.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:41:51.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:51 smithi149 ceph-mon[25685]: Cluster is now healthy 2024-03-24T11:41:51.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: failed to encode map e106 with expected crc 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: [24/Mar/2024:11:41:49] ENGINE Bus STARTING 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: [24/Mar/2024:11:41:50] ENGINE Serving on http://172.21.15.145:8765 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: [24/Mar/2024:11:41:50] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: [24/Mar/2024:11:41:50] ENGINE Bus STARTED 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:41:51.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: failed to encode map e106 with expected crc 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:49] ENGINE Bus STARTING 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:50] ENGINE Serving on http://172.21.15.145:8765 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:50] ENGINE Serving on https://172.21.15.145:7150 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: [24/Mar/2024:11:41:50] ENGINE Bus STARTED 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:41:51.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:51 smithi145 ceph-mon[29394]: Cluster is now healthy 2024-03-24T11:41:52.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:52 smithi145 ceph-mon[64662]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:52.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:52 smithi145 ceph-mon[64662]: mgrmap e36: y(active, since 3s), standbys: x 2024-03-24T11:41:52.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:52 smithi145 ceph-mon[29394]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:52.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:52 smithi145 ceph-mon[29394]: mgrmap e36: y(active, since 3s), standbys: x 2024-03-24T11:41:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:52 smithi149 ceph-mon[25685]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:52 smithi149 ceph-mon[25685]: mgrmap e36: y(active, since 3s), standbys: x 2024-03-24T11:41:54.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[64662]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:54.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[64662]: mgrmap e37: y(active, since 4s), standbys: x 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[29394]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[29394]: mgrmap e37: y(active, since 4s), standbys: x 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:54 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:54 smithi149 ceph-mon[25685]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:54 smithi149 ceph-mon[25685]: mgrmap e37: y(active, since 4s), standbys: x 2024-03-24T11:41:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:54 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:54 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:54 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:54 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:55.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:41:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:41:55] "GET /metrics HTTP/1.1" 200 35460 "" "Prometheus/2.33.4" 2024-03-24T11:41:56.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[64662]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:56 smithi145 ceph-mon[29394]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:56.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:56 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:56 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:56 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:56 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:41:56.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:56 smithi149 ceph-mon[25685]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:41:58.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:58 smithi145 ceph-mon[64662]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:41:58.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:58 smithi145 ceph-mon[29394]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:41:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:58 smithi149 ceph-mon[25685]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:41:58.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:41:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:41:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:42:00.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:42:00.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:42:00.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:41:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Unable to set osd_memory_target on smithi149 to 182481305: error parsing value: Value '182481305' is below minimum 939524096 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:42:00.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: Updating smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:41:59 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[29394]: Upgrade: It appears safe to stop mon.c 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[29394]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[64662]: Upgrade: It appears safe to stop mon.c 2024-03-24T11:42:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:00 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:01.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:00 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:01.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:00 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:01.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:00 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:42:01.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:00 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-24T11:42:01.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:00 smithi149 ceph-mon[25685]: Upgrade: It appears safe to stop mon.c 2024-03-24T11:42:01.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:00 smithi149 ceph-mon[25685]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:02.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[29394]: Upgrade: Updating mon.c 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[29394]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[29394]: Deploying daemon mon.c on smithi145 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[29394]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[64662]: Upgrade: Updating mon.c 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:02.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:02.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[64662]: Deploying daemon mon.c on smithi145 2024-03-24T11:42:02.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:02 smithi145 ceph-mon[64662]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:42:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:02 smithi149 ceph-mon[25685]: Upgrade: Updating mon.c 2024-03-24T11:42:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:02 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:02 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:02 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:02 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:02.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:02 smithi149 ceph-mon[25685]: Deploying daemon mon.c on smithi145 2024-03-24T11:42:02.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:02 smithi149 ceph-mon[25685]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:42:03.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:02 smithi145 systemd[1]: Stopping Ceph mon.c for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:42:03.508 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c[29389]: 2024-03-24T11:42:03.223+0000 7ff9560fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:42:03.509 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c[29389]: 2024-03-24T11:42:03.223+0000 7ff9560fb700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:42:03.509 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:03 smithi145 podman[66918]: 2024-03-24 11:42:03.265001539 +0000 UTC m=+0.158377323 container died 0b79dbdf9a4cb780600ffa1f4a90b9dc6198a5a3e77baf69565a072894029112 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.openshift.expose-services=, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-type=git, version=8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-24T11:42:03.783 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:03 smithi145 podman[66918]: 2024-03-24 11:42:03.599126096 +0000 UTC m=+0.492501914 container cleanup 0b79dbdf9a4cb780600ffa1f4a90b9dc6198a5a3e77baf69565a072894029112 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-24T11:42:03.784 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:03 smithi145 bash[66918]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c 2024-03-24T11:42:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:03 smithi145 podman[66932]: 2024-03-24 11:42:03.783808814 +0000 UTC m=+0.515474586 container remove 0b79dbdf9a4cb780600ffa1f4a90b9dc6198a5a3e77baf69565a072894029112 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, distribution-scope=public, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-24T11:42:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-24T11:42:04.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 systemd[1]: Stopped Ceph mon.c for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:42:04.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.c.service: Consumed 7.888s CPU time. 2024-03-24T11:42:04.563 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 systemd[1]: Starting Ceph mon.c for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:42:04.564 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 podman[67061]: 2024-03-24 11:42:04.33147837 +0000 UTC m=+0.018860547 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:42:04.564 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 podman[67061]: 2024-03-24 11:42:04.460060215 +0000 UTC m=+0.147442377 container create 4c983fe930c037b40829d97f2ede0f067586e2ca8f49d2d1bf5633b083f07e6a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T11:42:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 podman[67061]: 2024-03-24 11:42:04.679089116 +0000 UTC m=+0.366471291 container init 4c983fe930c037b40829d97f2ede0f067586e2ca8f49d2d1bf5633b083f07e6a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T11:42:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 podman[67061]: 2024-03-24 11:42:04.682627129 +0000 UTC m=+0.370009291 container start 4c983fe930c037b40829d97f2ede0f067586e2ca8f49d2d1bf5633b083f07e6a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T11:42:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: pidfile_write: ignore empty --pid-file 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: load: jerasure load: lrc 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Git sha 0 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: DB SUMMARY 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: DB Session ID: IBIP4CXWF3GK71P1GQB3 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:42:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 296801 ; 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.env: 0x5571a19e4ae0 2024-03-24T11:42:04.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.info_log: 0x5571a3cef8c0 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.statistics: (nil) 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.use_fsync: 0 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:42:04.865 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:42:04.866 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:42:04.866 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:42:04.866 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.db_log_dir: 2024-03-24T11:42:04.866 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.wal_dir: 2024-03-24T11:42:04.867 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:42:04.867 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:42:04.867 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:42:04.867 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.write_buffer_manager: 0x5571a3cc2dc0 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:42:04.868 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.unordered_write: 0 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.row_cache: None 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.wal_filter: None 2024-03-24T11:42:04.869 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:42:04.870 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:42:04.870 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:42:04.870 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:42:04.870 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.wal_compression: 0 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:42:04.871 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:42:04.872 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_open_files: -1 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Compression algorithms supported: 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kZSTD supported: 0 2024-03-24T11:42:04.873 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-24T11:42:04.874 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:42:04.875 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:42:04.875 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.merge_operator: 2024-03-24T11:42:04.875 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_filter: None 2024-03-24T11:42:04.875 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:42:04.875 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:42:04.875 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:42:04.875 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:42:04.876 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5571a3cef9c0) 2024-03-24T11:42:04.876 INFO:journalctl@ceph.mon.c.smithi145.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:42:04.876 INFO:journalctl@ceph.mon.c.smithi145.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:42:04.876 INFO:journalctl@ceph.mon.c.smithi145.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:42:04.876 INFO:journalctl@ceph.mon.c.smithi145.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:42:04.876 INFO:journalctl@ceph.mon.c.smithi145.stdout: index_type: 0 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: data_block_index_type: 0 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: index_shortening: 1 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: checksum: 4 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: no_block_cache: 0 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_cache: 0x5571a3cf3090 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_cache_options: 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: capacity : 536870912 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: num_shard_bits : 4 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: strict_capacity_limit : 0 2024-03-24T11:42:04.877 INFO:journalctl@ceph.mon.c.smithi145.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_cache_compressed: (nil) 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: persistent_cache: (nil) 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_size: 4096 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_size_deviation: 10 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_restart_interval: 16 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: index_block_restart_interval: 1 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: metadata_block_size: 4096 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: partition_filters: 0 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: use_delta_encoding: 1 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: filter_policy: bloomfilter 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: whole_key_filtering: 1 2024-03-24T11:42:04.878 INFO:journalctl@ceph.mon.c.smithi145.stdout: verify_compression: 0 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: format_version: 5 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: enable_index_compression: 1 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: block_align: 0 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: max_auto_readahead_size: 262144 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: prepopulate_block_cache: 0 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:42:04.879 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression: NoCompression 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.num_levels: 7 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:42:04.880 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:42:04.881 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:42:04.881 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:42:04.881 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:42:04.881 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:42:04.881 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:42:04.881 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:42:04.881 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:42:04.882 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:42:04.883 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:42:04.884 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:42:04.885 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.ttl: 2592000 2024-03-24T11:42:04.886 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:42:04.887 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15134, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b34fa237-15c7-4c85-92a7-9564df306674 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280524711880, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280524713279, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 197956, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15139, "largest_seqno": 15260, "table_properties": {"data_size": 196292, "index_size": 460, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 2062, "raw_average_key_size": 27, "raw_value_size": 194523, "raw_average_value_size": 2559, "num_data_blocks": 18, "num_entries": 76, "num_filter_entries": 76, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280524, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b34fa237-15c7-4c85-92a7-9564df306674", "db_session_id": "IBIP4CXWF3GK71P1GQB3", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280524713361, "job": 1, "event": "recovery_finished"} 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:42:04.888 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:42:04.890 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5571a3de4000 2024-03-24T11:42:04.890 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: DB pointer 0x5571a3dce000 2024-03-24T11:42:04.890 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:42:04.890 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:42:04.890 INFO:journalctl@ceph.mon.c.smithi145.stdout: ** DB Stats ** 2024-03-24T11:42:04.890 INFO:journalctl@ceph.mon.c.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: L0 1/0 193.32 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 223.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:42:04.891 INFO:journalctl@ceph.mon.c.smithi145.stdout: L6 1/0 11.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: Sum 2/0 11.62 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 223.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 223.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 223.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: 2024-03-24T11:42:04.892 INFO:journalctl@ceph.mon.c.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: Cumulative compaction: 0.00 GB write, 55.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: Interval compaction: 0.00 GB write, 55.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: Block cache BinnedLRUCache@0x5571a3cf3090#2 capacity: 512.00 MB usage: 0.78 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.55 KB,0.000104308%) Misc(1,0.00 KB,0%) 2024-03-24T11:42:04.893 INFO:journalctl@ceph.mon.c.smithi145.stdout: 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: starting mon.c rank 1 at public addrs [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] at bind addrs [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???) e3 preinit fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???).mds e1 new map 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???).mds e1 print_map 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout: e1 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout: legacy client fscid: -1 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout: 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout: No filesystems configured 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-24T11:42:04.894 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:42:04.895 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:42:04.895 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:42:04.895 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 ceph-mon[67080]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-03-24T11:42:04.895 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 bash[67061]: 4c983fe930c037b40829d97f2ede0f067586e2ca8f49d2d1bf5633b083f07e6a 2024-03-24T11:42:04.895 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:04 smithi145 systemd[1]: Started Ceph mon.c for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:42:05.818 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:42:05] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.33.4" 2024-03-24T11:42:06.098 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: mon.c calling monitor election 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: mon.a calling monitor election 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: fsmap 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: mgrmap e37: y(active, since 15s), standbys: x 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: overall HEALTH_OK 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:06.099 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[67080]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: mon.c calling monitor election 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: mon.a calling monitor election 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: fsmap 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: mgrmap e37: y(active, since 15s), standbys: x 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: overall HEALTH_OK 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.101 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:06.101 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.101 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:06 smithi145 ceph-mon[64662]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:42:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: mon.c calling monitor election 2024-03-24T11:42:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: mon.a calling monitor election 2024-03-24T11:42:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:42:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: fsmap 2024-03-24T11:42:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:06.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: mgrmap e37: y(active, since 15s), standbys: x 2024-03-24T11:42:06.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: overall HEALTH_OK 2024-03-24T11:42:06.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:06.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:06.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:06 smithi149 ceph-mon[25685]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:42:08.705 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:42:08.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:08 smithi145 ceph-mon[64662]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:42:08.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:08 smithi145 ceph-mon[67080]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:42:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:08 smithi149 ceph-mon[25685]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:42:08.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:42:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:42:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (2m) 15s ago 6m 16.4M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (86s) 15s ago 6m 45.4M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (55s) 15s ago 5m 50.5M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (45s) 15s ago 11m 446M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (80s) 15s ago 13m 473M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (30s) 15s ago 13m 29.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (11m) 15s ago 11m 52.9M 2048M 17.2.0 e1d6a67b021e 91790c8c1af1 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 starting - - - 2048M 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (2m) 15s ago 7m 17.0M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (102s) 15s ago 7m 18.3M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:42:09.386 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (11m) 15s ago 11m 51.5M 2736M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (10m) 15s ago 10m 53.4M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (10m) 15s ago 10m 48.3M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (9m) 15s ago 9m 51.4M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (9m) 15s ago 9m 55.2M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (9m) 15s ago 9m 50.0M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (8m) 15s ago 8m 49.1M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (8m) 15s ago 8m 52.2M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (50s) 15s ago 7m 53.0M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (6m) 15s ago 6m 70.4M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (6m) 15s ago 6m 74.5M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (6m) 15s ago 6m 71.4M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:42:09.387 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (5m) 15s ago 5m 73.4M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:42:09.723 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:09 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:42:09.721+0000 7f4ffb2c5700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:42:09.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:42:09.790 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr" 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:42:10.130 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "4/23 daemons upgraded", 2024-03-24T11:42:10.131 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading mon daemons", 2024-03-24T11:42:10.131 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:42:10.131 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:42:10.560 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:42:10.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2502486574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/835735499' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:10.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:10.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:10.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2502486574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:10.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:10 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/835735499' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:42:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:10.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/2502486574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:10.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:10 smithi149 ceph-mon[25685]: from='client.? 172.21.15.145:0/835735499' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:42:11.617 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[67080]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:11 smithi149 ceph-mon[25685]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:11 smithi149 ceph-mon[25685]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:11 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:11 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:12.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[64662]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:12.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:12.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:12.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:11 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:12 smithi149 ceph-mon[25685]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:13.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:12 smithi145 ceph-mon[64662]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:12 smithi145 ceph-mon[67080]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:14.961 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:14.961 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:14.961 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:14.961 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:14.961 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:14.961 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:14.962 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:14.962 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:14.962 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:42:14.962 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:14 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-24T11:42:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-24T11:42:15.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-24T11:42:15.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:15 smithi145 ceph-mon[64662]: Upgrade: It appears safe to stop mon.b 2024-03-24T11:42:15.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:42:15] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.33.4" 2024-03-24T11:42:15.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:15 smithi145 ceph-mon[67080]: Upgrade: It appears safe to stop mon.b 2024-03-24T11:42:15.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:15 smithi149 ceph-mon[25685]: Upgrade: It appears safe to stop mon.b 2024-03-24T11:42:16.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:16 smithi149 ceph-mon[25685]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:16.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:16 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:16.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:16 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:16.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:16 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:16.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:16 smithi149 ceph-mon[25685]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[67080]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:17.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:17.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:17.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[64662]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:17.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:17.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:17.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:17.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:16 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:17.714 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:17 smithi149 ceph-mon[25685]: Upgrade: Updating mon.b 2024-03-24T11:42:17.714 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:17 smithi149 ceph-mon[25685]: Deploying daemon mon.b on smithi149 2024-03-24T11:42:17.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:17 smithi149 systemd[1]: Stopping Ceph mon.b for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:42:18.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:17 smithi145 ceph-mon[64662]: Upgrade: Updating mon.b 2024-03-24T11:42:18.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:17 smithi145 ceph-mon[64662]: Deploying daemon mon.b on smithi149 2024-03-24T11:42:18.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:17 smithi145 ceph-mon[67080]: Upgrade: Updating mon.b 2024-03-24T11:42:18.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:17 smithi145 ceph-mon[67080]: Deploying daemon mon.b on smithi149 2024-03-24T11:42:18.392 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b[25667]: 2024-03-24T11:42:17.974+0000 7f9c713ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:42:18.392 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b[25667]: 2024-03-24T11:42:17.974+0000 7f9c713ec700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:42:18.392 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:18 smithi149 podman[57939]: 2024-03-24 11:42:18.138847209 +0000 UTC m=+0.282014013 container died 91790c8c1af1c6defad1cd55743c3eaa201d94023ed3f15101a09245042dbb23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, ceph=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, version=8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, architecture=x86_64, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=) 2024-03-24T11:42:18.658 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:18 smithi149 podman[57939]: 2024-03-24 11:42:18.499969479 +0000 UTC m=+0.643136287 container cleanup 91790c8c1af1c6defad1cd55743c3eaa201d94023ed3f15101a09245042dbb23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, distribution-scope=public, io.openshift.expose-services=, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-24T11:42:18.658 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:18 smithi149 bash[57939]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b 2024-03-24T11:42:18.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:18 smithi149 podman[57953]: 2024-03-24 11:42:18.78457452 +0000 UTC m=+0.642357582 container remove 91790c8c1af1c6defad1cd55743c3eaa201d94023ed3f15101a09245042dbb23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, distribution-scope=public, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-24T11:42:18.923 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:42:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:42:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:42:19.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:18 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-24T11:42:19.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:18 smithi149 systemd[1]: Stopped Ceph mon.b for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:42:19.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:18 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.b.service: Consumed 9.625s CPU time. 2024-03-24T11:42:19.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 systemd[1]: Starting Ceph mon.b for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:42:19.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 podman[58088]: 2024-03-24 11:42:19.24276673 +0000 UTC m=+0.019123416 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:42:19.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 podman[58088]: 2024-03-24 11:42:19.343946594 +0000 UTC m=+0.120303251 container create 6dedc9190b9eb1bb4c4ba2318c6d420f6c55e861d5fa9fe14f9c7453cca3c553 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-24T11:42:19.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 podman[58088]: 2024-03-24 11:42:19.571539037 +0000 UTC m=+0.347895705 container init 6dedc9190b9eb1bb4c4ba2318c6d420f6c55e861d5fa9fe14f9c7453cca3c553 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b) 2024-03-24T11:42:19.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 podman[58088]: 2024-03-24 11:42:19.574784979 +0000 UTC m=+0.351141640 container start 6dedc9190b9eb1bb4c4ba2318c6d420f6c55e861d5fa9fe14f9c7453cca3c553 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: pidfile_write: ignore empty --pid-file 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: load: jerasure load: lrc 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Git sha 0 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: DB SUMMARY 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: DB Session ID: FGT5GU7CVIICSY7W55JB 2024-03-24T11:42:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1045439 ; 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:42:19.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.env: 0x55b447aa3ae0 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.info_log: 0x55b44a12b8c0 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.statistics: (nil) 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.use_fsync: 0 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:42:19.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.db_log_dir: 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.wal_dir: 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:42:19.979 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.write_buffer_manager: 0x55b44a0fedc0 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.unordered_write: 0 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:42:19.980 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.row_cache: None 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.wal_filter: None 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.wal_compression: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:42:19.981 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:42:19.982 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:42:19.982 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:42:19.982 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:42:19.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_open_files: -1 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Compression algorithms supported: 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kZSTD supported: 0 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:42:19.984 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-24T11:42:19.985 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:42:19.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:42:19.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.merge_operator: 2024-03-24T11:42:19.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_filter: None 2024-03-24T11:42:19.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:42:19.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:42:19.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:42:19.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b44a12b9c0) 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_type: 0 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_index_type: 0 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_shortening: 1 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: checksum: 4 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: no_block_cache: 0 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache: 0x55b44a12f090 2024-03-24T11:42:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_options: 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: capacity : 536870912 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_shard_bits : 4 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: strict_capacity_limit : 0 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_compressed: (nil) 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: persistent_cache: (nil) 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size: 4096 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size_deviation: 10 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_restart_interval: 16 2024-03-24T11:42:19.988 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_block_restart_interval: 1 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: metadata_block_size: 4096 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: partition_filters: 0 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: use_delta_encoding: 1 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: filter_policy: bloomfilter 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: whole_key_filtering: 1 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: verify_compression: 0 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: format_version: 5 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_index_compression: 1 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_align: 0 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: max_auto_readahead_size: 262144 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: prepopulate_block_cache: 0 2024-03-24T11:42:19.989 INFO:journalctl@ceph.mon.b.smithi149.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression: NoCompression 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.num_levels: 7 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:42:19.990 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:42:19.991 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:42:19.992 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:42:19.993 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:42:19.994 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:42:19.994 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:42:19.994 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:42:19.994 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:42:19.994 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:42:19.994 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:42:19.994 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:42:19.995 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:42:19.996 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.ttl: 2592000 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:42:19.997 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15084, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9fc34c66-14fe-479e-abf1-ed35c7ba0dfb 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280539604610, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280539609010, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 626751, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15089, "largest_seqno": 15752, "table_properties": {"data_size": 623319, "index_size": 1265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1157, "raw_key_size": 8987, "raw_average_key_size": 20, "raw_value_size": 615759, "raw_average_value_size": 1393, "num_data_blocks": 57, "num_entries": 442, "num_filter_entries": 442, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280539, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9fc34c66-14fe-479e-abf1-ed35c7ba0dfb", "db_session_id": "FGT5GU7CVIICSY7W55JB", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280539609116, "job": 1, "event": "recovery_finished"} 2024-03-24T11:42:19.998 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b44a220000 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: DB pointer 0x55b44a20a000 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** DB Stats ** 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:42:19.999 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-24T11:42:20.000 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-03-24T11:42:20.000 INFO:journalctl@ceph.mon.b.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:42:20.000 INFO:journalctl@ceph.mon.b.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:42:20.000 INFO:journalctl@ceph.mon.b.smithi149.stdout: L0 1/0 612.06 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 258.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: L6 1/0 11.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: Sum 2/0 12.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 258.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 258.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 258.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-24T11:42:20.001 INFO:journalctl@ceph.mon.b.smithi149.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative compaction: 0.00 GB write, 64.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval compaction: 0.00 GB write, 64.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache BinnedLRUCache@0x55b44a12f090#2 capacity: 512.00 MB usage: 2.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,1.33 KB,0.00025332%) Misc(1,0.00 KB,0%) 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-24T11:42:20.002 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: starting mon.b rank 2 at public addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] at bind addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???) e3 preinit fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???).mds e1 new map 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???).mds e1 print_map 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout: e1 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout: legacy client fscid: -1 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout: No filesystems configured 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:42:20.003 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-24T11:42:20.004 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-03-24T11:42:20.004 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 bash[58088]: 6dedc9190b9eb1bb4c4ba2318c6d420f6c55e861d5fa9fe14f9c7453cca3c553 2024-03-24T11:42:20.004 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 systemd[1]: Started Ceph mon.b for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:42:20.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:19 smithi145 ceph-mon[64662]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:20.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:19 smithi145 ceph-mon[67080]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:20.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:19 smithi149 ceph-mon[58107]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:25.073 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:42:24.620+0000 7f4ffb2c5700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: mon.a calling monitor election 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: mon.c calling monitor election 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: monmap e4: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: fsmap 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: mgrmap e37: y(active, since 35s), standbys: x 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: mon.a calling monitor election 2024-03-24T11:42:25.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: mon.c calling monitor election 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: monmap e4: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: fsmap 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: mgrmap e37: y(active, since 35s), standbys: x 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:42:25.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-03-24T11:42:25.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:25.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:25.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:42:25] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.33.4" 2024-03-24T11:42:26.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:26 smithi145 ceph-mon[67080]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:26.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:26 smithi145 ceph-mon[64662]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:26.568 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: mon.a calling monitor election 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: mon.c calling monitor election 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: monmap e4: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: fsmap 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: mgrmap e37: y(active, since 35s), standbys: x 2024-03-24T11:42:26.569 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:42:26.570 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:42:26.570 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:42:26.570 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-03-24T11:42:26.570 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:26.570 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:26.570 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:26 smithi149 ceph-mon[58107]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: mon.b calling monitor election 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: mon.c calling monitor election 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: mon.a calling monitor election 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: monmap e4: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: fsmap 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: mgrmap e37: y(active, since 36s), standbys: x 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:42:27.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[64662]: overall HEALTH_OK 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: mon.b calling monitor election 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: mon.c calling monitor election 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: mon.a calling monitor election 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: monmap e4: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: fsmap 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: mgrmap e37: y(active, since 36s), standbys: x 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:42:27.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:27 smithi145 ceph-mon[67080]: overall HEALTH_OK 2024-03-24T11:42:27.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: mon.b calling monitor election 2024-03-24T11:42:27.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: mon.c calling monitor election 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: mon.a calling monitor election 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: monmap e4: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: fsmap 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: mgrmap e37: y(active, since 36s), standbys: x 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:42:27.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:27 smithi149 ceph-mon[58107]: overall HEALTH_OK 2024-03-24T11:42:28.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:28 smithi145 ceph-mon[64662]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:28.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:28 smithi145 ceph-mon[67080]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:28.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:28 smithi149 ceph-mon[58107]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:28.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:42:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:42:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:42:30.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:30 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:30.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:30 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:30.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:30 smithi145 ceph-mon[67080]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:30.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:30 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:30.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:30 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:30.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:30 smithi145 ceph-mon[64662]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:30 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:30 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:30 smithi149 ceph-mon[58107]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:31.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:31 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:31.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:31 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:31.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:31 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:31 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:31 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:31.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:31 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:31 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:31 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:31 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:32.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:32 smithi145 ceph-mon[64662]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:33.053 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:32 smithi145 ceph-mon[67080]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:33.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:32 smithi149 ceph-mon[58107]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:34.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:34.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:34.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:34.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:34.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:34.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:34.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:34.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:35.779 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:42:35] "GET /metrics HTTP/1.1" 200 38280 "" "Prometheus/2.33.4" 2024-03-24T11:42:35.779 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:35 smithi145 ceph-mon[67080]: Reconfiguring mon.a (monmap changed)... 2024-03-24T11:42:35.779 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:35 smithi145 ceph-mon[67080]: Reconfiguring daemon mon.a on smithi145 2024-03-24T11:42:35.779 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:35 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:35.780 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:35 smithi145 ceph-mon[64662]: Reconfiguring mon.a (monmap changed)... 2024-03-24T11:42:35.780 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:35 smithi145 ceph-mon[64662]: Reconfiguring daemon mon.a on smithi145 2024-03-24T11:42:35.780 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:35 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:35.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:35 smithi149 ceph-mon[58107]: Reconfiguring mon.a (monmap changed)... 2024-03-24T11:42:35.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:35 smithi149 ceph-mon[58107]: Reconfiguring daemon mon.a on smithi145 2024-03-24T11:42:35.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:35 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:36.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[67080]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:36.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:36.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[64662]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:36.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:36 smithi149 ceph-mon[58107]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:36.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:36.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:36.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:37 smithi149 ceph-mon[58107]: Reconfiguring mgr.y (monmap changed)... 2024-03-24T11:42:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:37 smithi149 ceph-mon[58107]: Reconfiguring daemon mgr.y on smithi145 2024-03-24T11:42:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:37.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:37.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:37.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:37.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[64662]: Reconfiguring mgr.y (monmap changed)... 2024-03-24T11:42:37.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[64662]: Reconfiguring daemon mgr.y on smithi145 2024-03-24T11:42:37.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:37.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:37.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:37.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:37.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:37.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[67080]: Reconfiguring mgr.y (monmap changed)... 2024-03-24T11:42:37.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[67080]: Reconfiguring daemon mgr.y on smithi145 2024-03-24T11:42:37.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:37.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:37.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:37.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:37.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:38 smithi149 ceph-mon[58107]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:42:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:38 smithi149 ceph-mon[58107]: Reconfiguring daemon mon.c on smithi145 2024-03-24T11:42:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:38 smithi149 ceph-mon[58107]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:38.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:42:38 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:42:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:42:39.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:38 smithi145 ceph-mon[64662]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:42:39.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:38 smithi145 ceph-mon[64662]: Reconfiguring daemon mon.c on smithi145 2024-03-24T11:42:39.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:38 smithi145 ceph-mon[64662]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:38 smithi145 ceph-mon[67080]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:42:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:38 smithi145 ceph-mon[67080]: Reconfiguring daemon mon.c on smithi145 2024-03-24T11:42:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:38 smithi145 ceph-mon[67080]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:39.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[64662]: Reconfiguring osd.0 (monmap changed)... 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.0 on smithi145 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[64662]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:39.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[67080]: Reconfiguring osd.0 (monmap changed)... 2024-03-24T11:42:39.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:42:39.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:39.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.0 on smithi145 2024-03-24T11:42:39.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:39 smithi145 ceph-mon[67080]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:42:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:39 smithi149 ceph-mon[58107]: Reconfiguring osd.0 (monmap changed)... 2024-03-24T11:42:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:42:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:40.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:39 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.0 on smithi145 2024-03-24T11:42:40.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:39 smithi149 ceph-mon[58107]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:42:40.914 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:42:41.580 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:42:41.580 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (2m) 31s ago 7m 17.7M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:42:41.580 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (118s) 12s ago 7m 45.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:42:41.580 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (87s) 31s ago 6m 51.7M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:42:41.580 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (77s) 12s ago 11m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (112s) 31s ago 13m 490M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (62s) 31s ago 13m 39.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (22s) 12s ago 12m 19.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (37s) 31s ago 12m 17.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (2m) 31s ago 8m 16.9M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (2m) 12s ago 8m 19.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (11m) 31s ago 11m 51.7M 2736M 17.2.0 e1d6a67b021e 58351dbda374 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (11m) 31s ago 11m 53.6M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (10m) 31s ago 10m 48.7M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (10m) 31s ago 10m 51.7M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (10m) 12s ago 10m 55.7M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (9m) 12s ago 9m 50.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (9m) 12s ago 9m 48.7M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (8m) 12s ago 8m 52.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:42:41.581 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (83s) 12s ago 8m 57.8M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:42:41.582 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (6m) 31s ago 6m 71.2M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:42:41.582 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (6m) 12s ago 6m 74.9M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:42:41.582 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (6m) 31s ago 6m 71.6M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:42:41.582 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (6m) 12s ago 6m 73.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:42:41.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:41 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:41.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:41 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:41.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:41 smithi149 ceph-mon[58107]: Reconfiguring osd.1 (monmap changed)... 2024-03-24T11:42:41.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:41 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:42:41.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:41 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:41.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:41 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.1 on smithi145 2024-03-24T11:42:41.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:41 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[67080]: Reconfiguring osd.1 (monmap changed)... 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.1 on smithi145 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:41.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:41.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:41.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[64662]: Reconfiguring osd.1 (monmap changed)... 2024-03-24T11:42:41.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:42:41.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:41.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.1 on smithi145 2024-03-24T11:42:41.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:41 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:41.979 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:42:41.979 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:42:41.979 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:42:41.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:41.979 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:42:41.979 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:42:41.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:42:41.980 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:42:42.360 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "5/23 daemons upgraded", 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading mon daemons", 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:42:42.361 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:42:42.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3608021148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:42.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3608021148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:42.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:42.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:42.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:42:42.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3608021148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:42:42.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:42.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:42.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:42:42.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:42.777 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:42:43.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[67080]: Reconfiguring osd.2 (monmap changed)... 2024-03-24T11:42:43.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.2 on smithi145 2024-03-24T11:42:43.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[67080]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:43.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/785994429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:42:43.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[64662]: Reconfiguring osd.2 (monmap changed)... 2024-03-24T11:42:43.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.2 on smithi145 2024-03-24T11:42:43.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[64662]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:43.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:43 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/785994429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:42:43.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:43 smithi149 ceph-mon[58107]: Reconfiguring osd.2 (monmap changed)... 2024-03-24T11:42:43.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:43 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.2 on smithi145 2024-03-24T11:42:43.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:43 smithi149 ceph-mon[58107]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:42:43.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:43 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/785994429' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:42:44.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[67080]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:42:44.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:44.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:44.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:42:44.693 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:44.693 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[64662]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:42:44.694 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:44.694 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:44.694 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:42:44.694 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:44 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:44.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:44 smithi149 ceph-mon[58107]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:42:44.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:44 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:44.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:44 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:44.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:44 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:42:44.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:44 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:45.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:42:45] "GET /metrics HTTP/1.1" 200 38280 "" "Prometheus/2.33.4" 2024-03-24T11:42:45.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[67080]: Reconfiguring osd.3 (monmap changed)... 2024-03-24T11:42:45.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.3 on smithi145 2024-03-24T11:42:45.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:45.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:45.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:45.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:45.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[64662]: Reconfiguring osd.3 (monmap changed)... 2024-03-24T11:42:45.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.3 on smithi145 2024-03-24T11:42:45.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:45.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:45.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:45.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:45 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:45.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:45 smithi149 ceph-mon[58107]: Reconfiguring osd.3 (monmap changed)... 2024-03-24T11:42:45.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:45 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.3 on smithi145 2024-03-24T11:42:45.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:45 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:45.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:45 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:45.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:45 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:45.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:45 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:46.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:46 smithi145 ceph-mon[67080]: Reconfiguring rgw.foo.smithi145.rqljdz (monmap changed)... 2024-03-24T11:42:46.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:46 smithi145 ceph-mon[67080]: Reconfiguring daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:42:46.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:46 smithi145 ceph-mon[67080]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:46 smithi145 ceph-mon[64662]: Reconfiguring rgw.foo.smithi145.rqljdz (monmap changed)... 2024-03-24T11:42:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:46 smithi145 ceph-mon[64662]: Reconfiguring daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:42:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:46 smithi145 ceph-mon[64662]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:46.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:46 smithi149 ceph-mon[58107]: Reconfiguring rgw.foo.smithi145.rqljdz (monmap changed)... 2024-03-24T11:42:46.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:46 smithi149 ceph-mon[58107]: Reconfiguring daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:42:46.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:46 smithi149 ceph-mon[58107]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:48.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[64662]: Reconfiguring rgw.smpl.smithi145.wrtpsu (monmap changed)... 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[64662]: Reconfiguring daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[64662]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:48.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[67080]: Reconfiguring rgw.smpl.smithi145.wrtpsu (monmap changed)... 2024-03-24T11:42:48.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:48.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:48.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[67080]: Reconfiguring daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:42:48.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:47 smithi145 ceph-mon[67080]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:47 smithi149 ceph-mon[58107]: Reconfiguring rgw.smpl.smithi145.wrtpsu (monmap changed)... 2024-03-24T11:42:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:47 smithi149 ceph-mon[58107]: Reconfiguring daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:42:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:47 smithi149 ceph-mon[58107]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:48.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:42:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:42:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[64662]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[64662]: Reconfiguring daemon mon.b on smithi149 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:49.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:49.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[67080]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:42:49.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:49.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:49.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:49.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:49 smithi145 ceph-mon[67080]: Reconfiguring daemon mon.b on smithi149 2024-03-24T11:42:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:49 smithi149 ceph-mon[58107]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:42:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:42:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:42:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:49 smithi149 ceph-mon[58107]: Reconfiguring daemon mon.b on smithi149 2024-03-24T11:42:50.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[64662]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[67080]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:50.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:50.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:50 smithi149 ceph-mon[58107]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:50.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:50.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:50.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:50.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:42:50.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:42:50.887 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:51.779 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:51 smithi149 ceph-mon[58107]: Reconfiguring mgr.x (monmap changed)... 2024-03-24T11:42:51.779 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:51 smithi149 ceph-mon[58107]: Reconfiguring daemon mgr.x on smithi149 2024-03-24T11:42:51.779 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:51 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:51.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:51 smithi145 ceph-mon[64662]: Reconfiguring mgr.x (monmap changed)... 2024-03-24T11:42:51.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:51 smithi145 ceph-mon[64662]: Reconfiguring daemon mgr.x on smithi149 2024-03-24T11:42:51.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:51 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:51.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:51 smithi145 ceph-mon[67080]: Reconfiguring mgr.x (monmap changed)... 2024-03-24T11:42:51.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:51 smithi145 ceph-mon[67080]: Reconfiguring daemon mgr.x on smithi149 2024-03-24T11:42:51.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:51 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:52.695 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:52 smithi149 ceph-mon[58107]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:52.695 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:52.695 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:52.695 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:42:52.695 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:52.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[64662]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[67080]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:42:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:53 smithi149 ceph-mon[58107]: Reconfiguring osd.4 (monmap changed)... 2024-03-24T11:42:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:53 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.4 on smithi149 2024-03-24T11:42:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:53 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:53.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:53 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:53.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[64662]: Reconfiguring osd.4 (monmap changed)... 2024-03-24T11:42:53.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.4 on smithi149 2024-03-24T11:42:53.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:53.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:53.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[67080]: Reconfiguring osd.4 (monmap changed)... 2024-03-24T11:42:53.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.4 on smithi149 2024-03-24T11:42:53.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:53.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:53 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:54 smithi149 ceph-mon[58107]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:54 smithi149 ceph-mon[58107]: Reconfiguring osd.5 (monmap changed)... 2024-03-24T11:42:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:54 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:42:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:54 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:54.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:54 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.5 on smithi149 2024-03-24T11:42:54.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[67080]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[67080]: Reconfiguring osd.5 (monmap changed)... 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.5 on smithi149 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[64662]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[64662]: Reconfiguring osd.5 (monmap changed)... 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:54 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.5 on smithi149 2024-03-24T11:42:55.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:42:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:42:55] "GET /metrics HTTP/1.1" 200 38281 "" "Prometheus/2.33.4" 2024-03-24T11:42:56.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:56.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:56.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:56 smithi149 ceph-mon[58107]: Reconfiguring osd.6 (monmap changed)... 2024-03-24T11:42:56.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:42:56.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:56.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:56 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.6 on smithi149 2024-03-24T11:42:56.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:56 smithi149 ceph-mon[58107]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:56.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:56.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:56.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[67080]: Reconfiguring osd.6 (monmap changed)... 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.6 on smithi149 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[67080]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[64662]: Reconfiguring osd.6 (monmap changed)... 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.6 on smithi149 2024-03-24T11:42:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:56 smithi145 ceph-mon[64662]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[64662]: Reconfiguring osd.7 (monmap changed)... 2024-03-24T11:42:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:42:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[64662]: Reconfiguring daemon osd.7 on smithi149 2024-03-24T11:42:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[64662]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:58.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:58.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:58.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[67080]: Reconfiguring osd.7 (monmap changed)... 2024-03-24T11:42:58.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:42:58.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:58.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[67080]: Reconfiguring daemon osd.7 on smithi149 2024-03-24T11:42:58.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:57 smithi145 ceph-mon[67080]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:57 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:57 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:57 smithi149 ceph-mon[58107]: Reconfiguring osd.7 (monmap changed)... 2024-03-24T11:42:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:57 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:42:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:57 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:57 smithi149 ceph-mon[58107]: Reconfiguring daemon osd.7 on smithi149 2024-03-24T11:42:58.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:57 smithi149 ceph-mon[58107]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:58.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:42:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:42:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:42:59.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[64662]: Reconfiguring rgw.foo.smithi149.gflbhe (monmap changed)... 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[64662]: Reconfiguring daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:59.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[67080]: Reconfiguring rgw.foo.smithi149.gflbhe (monmap changed)... 2024-03-24T11:42:59.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:59.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:59.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:42:59 smithi145 ceph-mon[67080]: Reconfiguring daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:42:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:59 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:59 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:42:59.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:59 smithi149 ceph-mon[58107]: Reconfiguring rgw.foo.smithi149.gflbhe (monmap changed)... 2024-03-24T11:42:59.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:59 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:42:59.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:59 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:59.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:42:59 smithi149 ceph-mon[58107]: Reconfiguring daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:43:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[67080]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:00.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[64662]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:43:00.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:00.949 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:00 smithi149 ceph-mon[58107]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:00.949 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:00.949 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:00.949 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:43:00.949 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:01.949 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:01 smithi149 ceph-mon[58107]: Reconfiguring rgw.smpl.smithi149.djutzp (monmap changed)... 2024-03-24T11:43:01.950 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:01 smithi149 ceph-mon[58107]: Reconfiguring daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:43:01.950 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:01 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:02.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:01 smithi145 ceph-mon[64662]: Reconfiguring rgw.smpl.smithi149.djutzp (monmap changed)... 2024-03-24T11:43:02.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:01 smithi145 ceph-mon[64662]: Reconfiguring daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:43:02.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:01 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:02.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:01 smithi145 ceph-mon[67080]: Reconfiguring rgw.smpl.smithi149.djutzp (monmap changed)... 2024-03-24T11:43:02.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:01 smithi145 ceph-mon[67080]: Reconfiguring daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:43:02.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:01 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.954 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.956 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-24T11:43:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-24T11:43:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-24T11:43:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-24T11:43:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-24T11:43:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:43:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:03 smithi149 ceph-mon[58107]: Upgrade: Setting container_image for all mon 2024-03-24T11:43:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:03 smithi149 ceph-mon[58107]: Upgrade: Setting container_image for all crash 2024-03-24T11:43:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:03 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:43:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:03 smithi149 ceph-mon[58107]: Upgrade: osd.0 is safe to restart 2024-03-24T11:43:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[67080]: Upgrade: Setting container_image for all mon 2024-03-24T11:43:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[67080]: Upgrade: Setting container_image for all crash 2024-03-24T11:43:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:43:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[67080]: Upgrade: osd.0 is safe to restart 2024-03-24T11:43:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[64662]: Upgrade: Setting container_image for all mon 2024-03-24T11:43:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[64662]: Upgrade: Setting container_image for all crash 2024-03-24T11:43:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-24T11:43:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:03 smithi145 ceph-mon[64662]: Upgrade: osd.0 is safe to restart 2024-03-24T11:43:04.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[67080]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:04.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:04.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:43:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[64662]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:43:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:04 smithi149 ceph-mon[58107]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:43:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:05.792 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:05 smithi145 ceph-mon[67080]: Upgrade: Updating osd.0 2024-03-24T11:43:05.792 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:05 smithi145 ceph-mon[67080]: Deploying daemon osd.0 on smithi145 2024-03-24T11:43:05.792 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:05 smithi145 ceph-mon[64662]: Upgrade: Updating osd.0 2024-03-24T11:43:05.792 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:05 smithi145 ceph-mon[64662]: Deploying daemon osd.0 on smithi145 2024-03-24T11:43:05.793 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:43:05] "GET /metrics HTTP/1.1" 200 38279 "" "Prometheus/2.33.4" 2024-03-24T11:43:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:05 smithi149 ceph-mon[58107]: Upgrade: Updating osd.0 2024-03-24T11:43:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:05 smithi149 ceph-mon[58107]: Deploying daemon osd.0 on smithi145 2024-03-24T11:43:06.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:06 smithi149 ceph-mon[58107]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:07.084 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:06 smithi145 ceph-mon[64662]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:07.084 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:06 smithi145 ceph-mon[67080]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:07.084 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:06 smithi145 systemd[1]: Stopping Ceph osd.0 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:43:07.361 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[32260]: 2024-03-24T11:43:07.082+0000 7ffbbef2c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:07.361 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[32260]: 2024-03-24T11:43:07.082+0000 7ffbbef2c700 -1 osd.0 106 *** Got signal Terminated *** 2024-03-24T11:43:07.361 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:07 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[32260]: 2024-03-24T11:43:07.082+0000 7ffbbef2c700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:43:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:07 smithi149 ceph-mon[58107]: osd.0 marked itself down and dead 2024-03-24T11:43:08.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:07 smithi145 ceph-mon[67080]: osd.0 marked itself down and dead 2024-03-24T11:43:08.015 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:07 smithi145 ceph-mon[64662]: osd.0 marked itself down and dead 2024-03-24T11:43:08.285 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:08 smithi145 podman[69977]: 2024-03-24 11:43:08.01508656 +0000 UTC m=+1.055542720 container died 58351dbda3747a2676502e6c4bd7f1509876cfdc3a7168ac68bbc08265cf0dab (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , release=754, vcs-type=git, io.openshift.expose-services=, io.buildah.version=1.19.8, name=centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-24T11:43:08.611 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:08 smithi145 podman[69977]: 2024-03-24 11:43:08.391594133 +0000 UTC m=+1.432050290 container cleanup 58351dbda3747a2676502e6c4bd7f1509876cfdc3a7168ac68bbc08265cf0dab (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, io.openshift.expose-services=, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, name=centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754) 2024-03-24T11:43:08.612 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:08 smithi145 bash[69977]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:43:08.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:43:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: failed to encode map e107 with expected crc 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: failed to encode map e107 with expected crc 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: osdmap e107: 8 total, 7 up, 8 in 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: failed to encode map e107 with expected crc 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: failed to encode map e107 with expected crc 2024-03-24T11:43:08.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:08 smithi149 ceph-mon[58107]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.012 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:08 smithi145 podman[69995]: 2024-03-24 11:43:08.651128151 +0000 UTC m=+0.627252408 container remove 58351dbda3747a2676502e6c4bd7f1509876cfdc3a7168ac68bbc08265cf0dab (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_CLEAN=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.expose-services=, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, architecture=x86_64, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, version=8, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8) 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: osdmap e107: 8 total, 7 up, 8 in 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[64662]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: osdmap e107: 8 total, 7 up, 8 in 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.013 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:08 smithi145 ceph-mon[67080]: failed to encode map e107 with expected crc 2024-03-24T11:43:09.263 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:09 smithi145 podman[70090]: 2024-03-24 11:43:08.914809269 +0000 UTC m=+0.017782593 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:09.264 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:09 smithi145 podman[70090]: 2024-03-24 11:43:09.03079124 +0000 UTC m=+0.133764592 container create 78a716191c5918a3d698162f001fa3c404386271b7fa0543cfda866c6bc4f16e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:43:09.559 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:09 smithi145 podman[70090]: 2024-03-24 11:43:09.263216009 +0000 UTC m=+0.366189330 container init 78a716191c5918a3d698162f001fa3c404386271b7fa0543cfda866c6bc4f16e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:43:09.559 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:09 smithi145 podman[70090]: 2024-03-24 11:43:09.266957742 +0000 UTC m=+0.369931062 container start 78a716191c5918a3d698162f001fa3c404386271b7fa0543cfda866c6bc4f16e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:43:09.560 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:09 smithi145 podman[70090]: 2024-03-24 11:43:09.36298193 +0000 UTC m=+0.465955244 container attach 78a716191c5918a3d698162f001fa3c404386271b7fa0543cfda866c6bc4f16e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:43:09.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:09 smithi145 podman[70090]: 2024-03-24 11:43:09.559949555 +0000 UTC m=+0.662922874 container died 78a716191c5918a3d698162f001fa3c404386271b7fa0543cfda866c6bc4f16e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: osdmap e108: 8 total, 7 up, 8 in 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[67080]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: osdmap e108: 8 total, 7 up, 8 in 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:09 smithi145 ceph-mon[64662]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: osdmap e108: 8 total, 7 up, 8 in 2024-03-24T11:43:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:09.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:09 smithi149 ceph-mon[58107]: failed to encode map e108 with expected crc 2024-03-24T11:43:10.860 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:10 smithi145 ceph-mon[67080]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:43:10.860 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:10 smithi145 ceph-mon[64662]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:43:10.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:10 smithi145 podman[70116]: 2024-03-24 11:43:10.442218867 +0000 UTC m=+0.877481950 container remove 78a716191c5918a3d698162f001fa3c404386271b7fa0543cfda866c6bc4f16e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T11:43:10.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:10 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T11:43:10.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:10 smithi145 systemd[1]: Stopped Ceph osd.0 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:43:10.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:10 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.0.service: Consumed 7.032s CPU time. 2024-03-24T11:43:10.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:10 smithi145 systemd[1]: Starting Ceph osd.0 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:43:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:10 smithi149 ceph-mon[58107]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:43:11.111 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:10 smithi145 podman[70219]: 2024-03-24 11:43:10.860019203 +0000 UTC m=+0.095453716 container create 8eac1de198adc750e37db37a287bc30b96d081232c1deaa6d6a540021e277a7c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:43:11.111 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:10 smithi145 podman[70219]: 2024-03-24 11:43:10.782481827 +0000 UTC m=+0.017916306 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:11.556 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 podman[70219]: 2024-03-24 11:43:11.115550226 +0000 UTC m=+0.350984694 container init 8eac1de198adc750e37db37a287bc30b96d081232c1deaa6d6a540021e277a7c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T11:43:11.556 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 podman[70219]: 2024-03-24 11:43:11.119057377 +0000 UTC m=+0.354491853 container start 8eac1de198adc750e37db37a287bc30b96d081232c1deaa6d6a540021e277a7c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:43:11.556 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 podman[70219]: 2024-03-24 11:43:11.212540978 +0000 UTC m=+0.447975446 container attach 8eac1de198adc750e37db37a287bc30b96d081232c1deaa6d6a540021e277a7c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b) 2024-03-24T11:43:11.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:11.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate[70235]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate[70235]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 bash[70219]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 bash[70219]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 bash[70219]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 bash[70219]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 bash[70219]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 bash[70219]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 bash[70219]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate[70235]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:43:11.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate[70235]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:43:11.863 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate[70235]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T11:43:11.863 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate[70235]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T11:43:11.863 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate[70235]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T11:43:11.863 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:11 smithi145 podman[70219]: 2024-03-24 11:43:11.581182164 +0000 UTC m=+0.816616634 container died 8eac1de198adc750e37db37a287bc30b96d081232c1deaa6d6a540021e277a7c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:43:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:11 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:12.591 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 podman[70356]: 2024-03-24 11:43:12.258224274 +0000 UTC m=+0.671806002 container remove 8eac1de198adc750e37db37a287bc30b96d081232c1deaa6d6a540021e277a7c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T11:43:12.592 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 podman[70421]: 2024-03-24 11:43:12.372100011 +0000 UTC m=+0.018074772 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:12.592 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 podman[70421]: 2024-03-24 11:43:12.475912456 +0000 UTC m=+0.121887245 container create b5960e0232afd0815c23f5d2829dc8acc52dc0156c58f6dae573ec809a9c1d1e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T11:43:12.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:12 smithi145 ceph-mon[67080]: pgmap v46: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:12.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:12 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:12.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:12 smithi145 ceph-mon[64662]: pgmap v46: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:12.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:12 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:12.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 podman[70421]: 2024-03-24 11:43:12.695207127 +0000 UTC m=+0.341181888 container init b5960e0232afd0815c23f5d2829dc8acc52dc0156c58f6dae573ec809a9c1d1e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T11:43:12.862 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 podman[70421]: 2024-03-24 11:43:12.698563168 +0000 UTC m=+0.344537916 container start b5960e0232afd0815c23f5d2829dc8acc52dc0156c58f6dae573ec809a9c1d1e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:43:12.863 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 ceph-osd[70440]: -- 172.21.15.145:0/3194092482 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f19133d1e0 con 0x55f19130ec00 2024-03-24T11:43:12.863 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 bash[70421]: b5960e0232afd0815c23f5d2829dc8acc52dc0156c58f6dae573ec809a9c1d1e 2024-03-24T11:43:12.863 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:12 smithi145 systemd[1]: Started Ceph osd.0 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:43:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:12 smithi149 ceph-mon[58107]: pgmap v46: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:12 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 21 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:13.153 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:43:13.611 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[70436]: 2024-03-24T11:43:13.244+0000 7f756dfb67c0 -1 Falling back to public interface 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (3m) 64s ago 8m 17.7M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (2m) 44s ago 7m 45.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (2m) 64s ago 6m 51.7M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (109s) 44s ago 12m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (2m) 64s ago 14m 490M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (94s) 64s ago 14m 39.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:43:13.834 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (54s) 44s ago 12m 19.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (69s) 64s ago 12m 17.7M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (3m) 64s ago 8m 16.9M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (2m) 44s ago 8m 19.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 starting - - - 2736M 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (11m) 64s ago 11m 53.6M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (11m) 64s ago 11m 48.7M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (11m) 64s ago 11m 51.7M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (10m) 44s ago 10m 55.7M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (10m) 44s ago 10m 50.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (9m) 44s ago 9m 48.7M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (9m) 44s ago 9m 52.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (115s) 44s ago 8m 57.8M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (7m) 64s ago 7m 71.2M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:43:13.835 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (7m) 44s ago 7m 74.9M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:43:13.836 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (7m) 64s ago 7m 71.6M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:43:13.836 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (7m) 44s ago 7m 73.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:43:14.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:14.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:14.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[67080]: from='client.44152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[67080]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[67080]: pgmap v47: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-03-24T11:43:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[64662]: from='client.44152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[64662]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:13 smithi145 ceph-mon[64662]: pgmap v47: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-03-24T11:43:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:13 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:13 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:13 smithi149 ceph-mon[58107]: from='client.44152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:13 smithi149 ceph-mon[58107]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:13 smithi149 ceph-mon[58107]: pgmap v47: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-03-24T11:43:14.238 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:43:14.238 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:43:14.238 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:43:14.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:14.238 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:43:14.238 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:43:14.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:43:14.239 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:43:14.592 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "5/23 daemons upgraded", 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:43:14.593 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:43:15.005 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 71/627 objects degraded (11.324%), 21 pgs degraded 2024-03-24T11:43:15.005 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T11:43:15.005 INFO:teuthology.orchestra.run.smithi145.stdout: osd.0 (root=default,host=smithi145) is down 2024-03-24T11:43:15.005 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 71/627 objects degraded (11.324%), 21 pgs degraded 2024-03-24T11:43:15.005 INFO:teuthology.orchestra.run.smithi145.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-03-24T11:43:15.005 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-03-24T11:43:15.005 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.19 is active+undersized+degraded, acting [3,6] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-03-24T11:43:15.006 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-03-24T11:43:15.007 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-03-24T11:43:15.007 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-03-24T11:43:15.007 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-03-24T11:43:15.007 INFO:teuthology.orchestra.run.smithi145.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-03-24T11:43:15.007 INFO:teuthology.orchestra.run.smithi145.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-03-24T11:43:15.007 INFO:teuthology.orchestra.run.smithi145.stdout: pg 5.1e is active+undersized+degraded, acting [7,2] 2024-03-24T11:43:15.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:14 smithi145 ceph-mon[64662]: from='client.44164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:15.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:14 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/921208878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:15.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:14 smithi145 ceph-mon[64662]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:14 smithi145 ceph-mon[67080]: from='client.44164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:14 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/921208878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:14 smithi145 ceph-mon[67080]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:14 smithi149 ceph-mon[58107]: from='client.44164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:14 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/921208878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:15.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:14 smithi149 ceph-mon[58107]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:15.860 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:43:15] "GET /metrics HTTP/1.1" 200 38279 "" "Prometheus/2.33.4" 2024-03-24T11:43:16.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:15 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/84021667' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:43:16.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:15 smithi145 ceph-mon[67080]: pgmap v48: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:16.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:15 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/84021667' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:43:16.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:15 smithi145 ceph-mon[64662]: pgmap v48: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:16.112 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[70436]: 2024-03-24T11:43:15.977+0000 7f756dfb67c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-24T11:43:16.112 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[70436]: 2024-03-24T11:43:16.008+0000 7f756dfb67c0 -1 osd.0 106 log_to_monitors true 2024-03-24T11:43:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:15 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/84021667' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:43:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:15 smithi149 ceph-mon[58107]: pgmap v48: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:17.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:16 smithi145 ceph-mon[64662]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:43:17.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:16 smithi145 ceph-mon[67080]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:43:17.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:16 smithi149 ceph-mon[58107]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:43:17.611 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[70436]: 2024-03-24T11:43:17.341+0000 7f755b406700 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:43:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:43:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: osdmap e109: 8 total, 7 up, 8 in 2024-03-24T11:43:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:43:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' 2024-03-24T11:43:18.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:18.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:18.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:17 smithi149 ceph-mon[58107]: pgmap v50: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 459 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:18.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: osdmap e109: 8 total, 7 up, 8 in 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[67080]: pgmap v50: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 459 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:18.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: osdmap e109: 8 total, 7 up, 8 in 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: failed to encode map e109 with expected crc 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: from='osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406]' entity='osd.0' 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:18.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:17 smithi145 ceph-mon[64662]: pgmap v50: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 459 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:18.977 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:43:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:43:18.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:43:18.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: failed to encode map e110 with expected crc 2024-03-24T11:43:18.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406] boot 2024-03-24T11:43:18.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:43:18.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:43:18.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: failed to encode map e110 with expected crc 2024-03-24T11:43:18.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:18.978 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:19.139 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: failed to encode map e110 with expected crc 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406] boot 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: failed to encode map e110 with expected crc 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: failed to encode map e110 with expected crc 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: osd.0 [v2:172.21.15.145:6802/1536583406,v1:172.21.15.145:6803/1536583406] boot 2024-03-24T11:43:19.140 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:43:19.141 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:43:19.141 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: failed to encode map e110 with expected crc 2024-03-24T11:43:19.141 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:19.141 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:20.167 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e110 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e110 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: pgmap v53: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:20.168 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e110 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e110 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: pgmap v53: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:20.169 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e110 with expected crc 2024-03-24T11:43:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e110 with expected crc 2024-03-24T11:43:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:43:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: pgmap v53: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:43:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:20.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:20 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:20 smithi145 ceph-mon[64662]: failed to encode map e111 with expected crc 2024-03-24T11:43:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:20 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:20 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:20 smithi145 ceph-mon[67080]: failed to encode map e111 with expected crc 2024-03-24T11:43:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:20 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:21.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:20 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:21.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:20 smithi149 ceph-mon[58107]: failed to encode map e111 with expected crc 2024-03-24T11:43:21.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:20 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:21.962 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:21 smithi145 ceph-mon[67080]: pgmap v54: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-24T11:43:21.963 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:21 smithi145 ceph-mon[64662]: pgmap v54: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-24T11:43:22.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:21 smithi149 ceph-mon[58107]: pgmap v54: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-24T11:43:22.361 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:43:22.169+0000 7f4ffb2c5700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-24T11:43:23.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:23.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:23.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:22 smithi149 ceph-mon[58107]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-24T11:43:23.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:23.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[67080]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:23.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:23.364 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:23.364 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:22 smithi145 ceph-mon[64662]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-24T11:43:24.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:23 smithi149 ceph-mon[58107]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 0 op/s 2024-03-24T11:43:24.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:23 smithi145 ceph-mon[67080]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 0 op/s 2024-03-24T11:43:24.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:23 smithi145 ceph-mon[64662]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 0 op/s 2024-03-24T11:43:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:24 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-03-24T11:43:25.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:24 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:43:25.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:24 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-03-24T11:43:25.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:24 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:43:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:24 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-03-24T11:43:25.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:24 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:43:25.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:43:25] "GET /metrics HTTP/1.1" 200 38213 "" "Prometheus/2.33.4" 2024-03-24T11:43:26.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:25 smithi149 ceph-mon[58107]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:26.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:25 smithi145 ceph-mon[67080]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:26.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:25 smithi145 ceph-mon[64662]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:28.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:28 smithi145 ceph-mon[67080]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:43:28.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:28 smithi145 ceph-mon[64662]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:43:28.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:28 smithi149 ceph-mon[58107]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:43:28.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:43:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:43:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:43:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:30 smithi149 ceph-mon[58107]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-24T11:43:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:30 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:31.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:30 smithi145 ceph-mon[67080]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-24T11:43:31.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:30 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:30 smithi145 ceph-mon[64662]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-24T11:43:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:30 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:31 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:32.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:31 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:31 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:32.941 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:32 smithi149 ceph-mon[58107]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:33.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:32 smithi145 ceph-mon[64662]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:33.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:32 smithi145 ceph-mon[67080]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:34 smithi149 ceph-mon[58107]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:35.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:34 smithi145 ceph-mon[64662]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:35.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:34 smithi145 ceph-mon[67080]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:35.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:35.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:35.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:43:35] "GET /metrics HTTP/1.1" 200 38351 "" "Prometheus/2.33.4" 2024-03-24T11:43:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:36 smithi149 ceph-mon[58107]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:37.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:36 smithi145 ceph-mon[64662]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:37.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:36 smithi145 ceph-mon[67080]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:37.964 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:37.964 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:38.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:38 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:38.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:38 smithi145 ceph-mon[64662]: Upgrade: osd.1 is safe to restart 2024-03-24T11:43:38.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:38 smithi145 ceph-mon[64662]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:38 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:38 smithi145 ceph-mon[67080]: Upgrade: osd.1 is safe to restart 2024-03-24T11:43:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:38 smithi145 ceph-mon[67080]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:38 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-24T11:43:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:38 smithi149 ceph-mon[58107]: Upgrade: osd.1 is safe to restart 2024-03-24T11:43:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:38 smithi149 ceph-mon[58107]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:38.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:43:38 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:43:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:43:40.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[64662]: Upgrade: Updating osd.1 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[64662]: Deploying daemon osd.1 on smithi145 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[64662]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[67080]: Upgrade: Updating osd.1 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:43:40.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[67080]: Deploying daemon osd.1 on smithi145 2024-03-24T11:43:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:39 smithi145 ceph-mon[67080]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:39 smithi149 ceph-mon[58107]: Upgrade: Updating osd.1 2024-03-24T11:43:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:43:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:39 smithi149 ceph-mon[58107]: Deploying daemon osd.1 on smithi145 2024-03-24T11:43:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:39 smithi149 ceph-mon[58107]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:41.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:40 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:41.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:40 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:41.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:40 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:42.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:41 smithi145 ceph-mon[67080]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:42.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:41 smithi145 ceph-mon[64662]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:42.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:41 smithi149 ceph-mon[58107]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:42.453 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:42 smithi145 systemd[1]: Stopping Ceph osd.1 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:43:42.861 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[35228]: 2024-03-24T11:43:42.452+0000 7f3799a81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:42.861 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[35228]: 2024-03-24T11:43:42.452+0000 7f3799a81700 -1 osd.1 111 *** Got signal Terminated *** 2024-03-24T11:43:42.861 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[35228]: 2024-03-24T11:43:42.452+0000 7f3799a81700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:43:43.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:42 smithi149 ceph-mon[58107]: osd.1 marked itself down and dead 2024-03-24T11:43:43.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:42 smithi145 ceph-mon[64662]: osd.1 marked itself down and dead 2024-03-24T11:43:43.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:42 smithi145 ceph-mon[67080]: osd.1 marked itself down and dead 2024-03-24T11:43:43.779 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:43 smithi145 podman[72458]: 2024-03-24 11:43:43.485945709 +0000 UTC m=+1.153480051 container died 66d2dfac49ba29e00befd48c107746cf634ccb8ccbb806ba01fef523d0567b7d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, ceph=True, architecture=x86_64) 2024-03-24T11:43:44.111 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:43 smithi145 podman[72458]: 2024-03-24 11:43:43.885303283 +0000 UTC m=+1.552837626 container cleanup 66d2dfac49ba29e00befd48c107746cf634ccb8ccbb806ba01fef523d0567b7d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, ceph=True, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, build-date=2022-05-03T08:36:31.336870, release=754, GIT_CLEAN=True, io.openshift.expose-services=, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-type=git, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-24T11:43:44.111 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:43 smithi145 bash[72458]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: osdmap e112: 8 total, 7 up, 8 in 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[64662]: pgmap v66: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: osdmap e112: 8 total, 7 up, 8 in 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:43 smithi145 ceph-mon[67080]: pgmap v66: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:43:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: osdmap e112: 8 total, 7 up, 8 in 2024-03-24T11:43:44.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: failed to encode map e112 with expected crc 2024-03-24T11:43:44.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:43 smithi149 ceph-mon[58107]: pgmap v66: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:44.513 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:44 smithi145 podman[72476]: 2024-03-24 11:43:44.163358973 +0000 UTC m=+0.675920970 container remove 66d2dfac49ba29e00befd48c107746cf634ccb8ccbb806ba01fef523d0567b7d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, name=centos-stream, version=8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, RELEASE=HEAD, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD) 2024-03-24T11:43:44.766 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:44 smithi145 podman[72569]: 2024-03-24 11:43:44.512996448 +0000 UTC m=+0.112506515 container create 1d16d77a83a4c391860016005e263c831aea32a6f24be9ef0cb2c94cd4e87640 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:43:44.767 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:44 smithi145 podman[72569]: 2024-03-24 11:43:44.418409678 +0000 UTC m=+0.017919733 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:45.081 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:44 smithi145 podman[72569]: 2024-03-24 11:43:44.76663373 +0000 UTC m=+0.366143759 container init 1d16d77a83a4c391860016005e263c831aea32a6f24be9ef0cb2c94cd4e87640 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T11:43:45.081 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:44 smithi145 podman[72569]: 2024-03-24 11:43:44.770378028 +0000 UTC m=+0.369888058 container start 1d16d77a83a4c391860016005e263c831aea32a6f24be9ef0cb2c94cd4e87640 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:43:45.082 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:44 smithi145 podman[72569]: 2024-03-24 11:43:44.848811709 +0000 UTC m=+0.448321739 container attach 1d16d77a83a4c391860016005e263c831aea32a6f24be9ef0cb2c94cd4e87640 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: osdmap e113: 8 total, 7 up, 8 in 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.082 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[64662]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: osdmap e113: 8 total, 7 up, 8 in 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.083 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.084 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.084 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.084 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.084 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:44 smithi145 ceph-mon[67080]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.360 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:45 smithi145 podman[72569]: 2024-03-24 11:43:45.081602961 +0000 UTC m=+0.681112993 container died 1d16d77a83a4c391860016005e263c831aea32a6f24be9ef0cb2c94cd4e87640 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T11:43:45.360 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:43:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: osdmap e113: 8 total, 7 up, 8 in 2024-03-24T11:43:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:44 smithi149 ceph-mon[58107]: failed to encode map e113 with expected crc 2024-03-24T11:43:45.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:43:45] "GET /metrics HTTP/1.1" 200 38351 "" "Prometheus/2.33.4" 2024-03-24T11:43:46.034 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:45 smithi145 ceph-mon[67080]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:46.035 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:45 smithi145 ceph-mon[67080]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 1 stale+active+clean, 34 active+undersized, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:43:46.035 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:45 smithi145 podman[72598]: 2024-03-24 11:43:45.778201754 +0000 UTC m=+0.690548521 container remove 1d16d77a83a4c391860016005e263c831aea32a6f24be9ef0cb2c94cd4e87640 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:43:46.035 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:45 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-24T11:43:46.035 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:45 smithi145 systemd[1]: Stopped Ceph osd.1 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:43:46.035 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:45 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.1.service: Consumed 7.648s CPU time. 2024-03-24T11:43:46.035 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:46 smithi145 systemd[1]: Starting Ceph osd.1 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:43:46.036 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:45 smithi145 ceph-mon[64662]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:46.036 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:45 smithi145 ceph-mon[64662]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 1 stale+active+clean, 34 active+undersized, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (3m) 28s ago 8m 26.2M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (3m) 76s ago 8m 45.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (2m) 28s ago 7m 54.7M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (2m) 76s ago 13m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (2m) 28s ago 14m 498M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (2m) 28s ago 14m 47.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (86s) 76s ago 13m 19.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (101s) 28s ago 13m 38.5M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:43:46.036 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (3m) 28s ago 9m 17.3M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (3m) 76s ago 9m 19.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (33s) 28s ago 12m 12.6M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (12m) 28s ago 12m 54.7M 2736M 17.2.0 e1d6a67b021e 66d2dfac49ba 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (11m) 28s ago 11m 49.5M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (11m) 28s ago 11m 53.2M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (11m) 76s ago 11m 55.7M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (10m) 76s ago 10m 50.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (10m) 76s ago 10m 48.7M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (9m) 76s ago 9m 52.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (2m) 76s ago 9m 57.8M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (7m) 28s ago 7m 71.5M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (7m) 76s ago 7m 74.9M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (7m) 28s ago 7m 72.2M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:43:46.037 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (7m) 76s ago 7m 73.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:43:46.346 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:46 smithi145 podman[72771]: 2024-03-24 11:43:46.137719107 +0000 UTC m=+0.024427735 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:46.346 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:46 smithi145 podman[72771]: 2024-03-24 11:43:46.246492932 +0000 UTC m=+0.133201559 container create c4deae20ef96087574e8eb9acfa638992db89ad65305fec902cb826ec95feef0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:43:46.456 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:43:46.456 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:43:46.456 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:43:46.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:46.456 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 1 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 6 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:43:46.457 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:43:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:45 smithi149 ceph-mon[58107]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:45 smithi149 ceph-mon[58107]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 1 stale+active+clean, 34 active+undersized, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:43:46.611 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:46 smithi145 podman[72771]: 2024-03-24 11:43:46.590827403 +0000 UTC m=+0.477536041 container init c4deae20ef96087574e8eb9acfa638992db89ad65305fec902cb826ec95feef0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:43:46.612 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:46 smithi145 podman[72771]: 2024-03-24 11:43:46.595211192 +0000 UTC m=+0.481919827 container start c4deae20ef96087574e8eb9acfa638992db89ad65305fec902cb826ec95feef0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "6/23 daemons upgraded", 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:43:46.793 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:43:46.794 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:43:46.993 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:46 smithi145 podman[72771]: 2024-03-24 11:43:46.69932414 +0000 UTC m=+0.586032769 container attach c4deae20ef96087574e8eb9acfa638992db89ad65305fec902cb826ec95feef0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 82/627 objects degraded (13.078%), 22 pgs degraded 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout: osd.1 (root=default,host=smithi145) is down 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 82/627 objects degraded (13.078%), 22 pgs degraded 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-03-24T11:43:47.232 INFO:teuthology.orchestra.run.smithi145.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 4.d is active+undersized+degraded, acting [4,2] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 4.e is active+undersized+degraded, acting [4,6] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-03-24T11:43:47.233 INFO:teuthology.orchestra.run.smithi145.stdout: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-03-24T11:43:47.234 INFO:teuthology.orchestra.run.smithi145.stdout: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-03-24T11:43:47.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[67080]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:47.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 82/627 objects degraded (13.078%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:47.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[67080]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:47.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2698912902' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:47.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[64662]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:47.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 82/627 objects degraded (13.078%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:47.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[64662]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:47.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:46 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2698912902' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate[72805]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate[72805]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 bash[72771]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 bash[72771]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 bash[72771]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 bash[72771]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 bash[72771]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 bash[72771]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 bash[72771]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate[72805]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate[72805]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate[72805]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-24T11:43:47.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate[72805]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T11:43:47.364 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate[72805]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-24T11:43:47.364 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 podman[72771]: 2024-03-24 11:43:47.091065633 +0000 UTC m=+0.977774270 container died c4deae20ef96087574e8eb9acfa638992db89ad65305fec902cb826ec95feef0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:43:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:46 smithi149 ceph-mon[58107]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:46 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 82/627 objects degraded (13.078%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:46 smithi149 ceph-mon[58107]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:46 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2698912902' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:47.998 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 podman[72983]: 2024-03-24 11:43:47.745131563 +0000 UTC m=+0.648698160 container remove c4deae20ef96087574e8eb9acfa638992db89ad65305fec902cb826ec95feef0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T11:43:47.998 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 podman[73049]: 2024-03-24 11:43:47.948459116 +0000 UTC m=+0.106019663 container create 848b03020bb19af4f4fdcaad341455ca1ce983c598d3a7db61a1643f6db49802 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True) 2024-03-24T11:43:47.998 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:47 smithi145 podman[73049]: 2024-03-24 11:43:47.860053112 +0000 UTC m=+0.017613668 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:43:48.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-mon[67080]: from='client.44215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:48.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3217671136' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:43:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-mon[67080]: pgmap v69: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-mon[64662]: from='client.44215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3217671136' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:43:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:47 smithi145 ceph-mon[64662]: pgmap v69: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:48.362 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:48 smithi145 podman[73049]: 2024-03-24 11:43:48.143242134 +0000 UTC m=+0.300802692 container init 848b03020bb19af4f4fdcaad341455ca1ce983c598d3a7db61a1643f6db49802 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T11:43:48.362 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:48 smithi145 podman[73049]: 2024-03-24 11:43:48.146596966 +0000 UTC m=+0.304157515 container start 848b03020bb19af4f4fdcaad341455ca1ce983c598d3a7db61a1643f6db49802 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b) 2024-03-24T11:43:48.362 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:48 smithi145 ceph-osd[73068]: -- 172.21.15.145:0/1068079757 <== mon.1 v2:172.21.15.145:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dbbc7611e0 con 0x55dbbc733400 2024-03-24T11:43:48.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:48 smithi145 bash[73049]: 848b03020bb19af4f4fdcaad341455ca1ce983c598d3a7db61a1643f6db49802 2024-03-24T11:43:48.363 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:48 smithi145 systemd[1]: Started Ceph osd.1 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:43:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:48 smithi149 ceph-mon[58107]: from='client.44215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:43:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:48 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3217671136' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:43:48.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:48 smithi149 ceph-mon[58107]: pgmap v69: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:48.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:43:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:43:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:43:49.112 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:48 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[73064]: 2024-03-24T11:43:48.698+0000 7f2e68c917c0 -1 Falling back to public interface 2024-03-24T11:43:49.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:49.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:49.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:49.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:49.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:49.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:50.949 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:50 smithi145 ceph-mon[64662]: pgmap v70: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:50.949 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:50.949 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:50.950 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:50 smithi145 ceph-mon[67080]: pgmap v70: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:50.950 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:50.950 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:50 smithi149 ceph-mon[58107]: pgmap v70: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:51.771 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:51 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:51.772 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:51 smithi145 ceph-mon[67080]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:43:51.772 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:51 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:51.772 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:51 smithi145 ceph-mon[64662]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:43:51.772 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[73064]: 2024-03-24T11:43:51.405+0000 7f2e68c917c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-24T11:43:51.772 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[73064]: 2024-03-24T11:43:51.436+0000 7f2e68c917c0 -1 osd.1 111 log_to_monitors true 2024-03-24T11:43:51.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:51 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:51.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:51 smithi149 ceph-mon[58107]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:43:52.361 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[73064]: 2024-03-24T11:43:52.095+0000 7f2e560e1700 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:43:52.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: pgmap v71: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: osdmap e114: 8 total, 7 up, 8 in 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[64662]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: pgmap v71: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:43:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: osdmap e114: 8 total, 7 up, 8 in 2024-03-24T11:43:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:43:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' 2024-03-24T11:43:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:52 smithi145 ceph-mon[67080]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: pgmap v71: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-03-24T11:43:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:43:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: failed to encode map e114 with expected crc 2024-03-24T11:43:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: osdmap e114: 8 total, 7 up, 8 in 2024-03-24T11:43:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:43:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: from='osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038]' entity='osd.1' 2024-03-24T11:43:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:52 smithi149 ceph-mon[58107]: failed to encode map e114 with expected crc 2024-03-24T11:43:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: failed to encode map e114 with expected crc 2024-03-24T11:43:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: failed to encode map e114 with expected crc 2024-03-24T11:43:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:43:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: failed to encode map e115 with expected crc 2024-03-24T11:43:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038] boot 2024-03-24T11:43:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:43:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: failed to encode map e115 with expected crc 2024-03-24T11:43:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:53 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:43:54.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: failed to encode map e114 with expected crc 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: failed to encode map e114 with expected crc 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038] boot 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:43:54.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: failed to encode map e114 with expected crc 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: failed to encode map e114 with expected crc 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: osd.1 [v2:172.21.15.145:6810/3545843038,v1:172.21.15.145:6811/3545843038] boot 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:53 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: pgmap v74: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 88/627 objects degraded (14.035%) 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 88/627 objects degraded (14.035%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: pgmap v74: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 88/627 objects degraded (14.035%) 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.864 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:54 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 88/627 objects degraded (14.035%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e115 with expected crc 2024-03-24T11:43:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: pgmap v74: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 88/627 objects degraded (14.035%) 2024-03-24T11:43:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:54 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 88/627 objects degraded (14.035%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:43:55.821 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:43:55] "GET /metrics HTTP/1.1" 200 38421 "" "Prometheus/2.33.4" 2024-03-24T11:43:55.821 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.821 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[64662]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.822 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:55 smithi145 ceph-mon[67080]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:55 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:55 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:55 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:55 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:55.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:55 smithi149 ceph-mon[58107]: failed to encode map e116 with expected crc 2024-03-24T11:43:56.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:56 smithi145 ceph-mon[64662]: pgmap v76: 161 pgs: 8 active+undersized+degraded, 17 active+undersized, 136 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 39/627 objects degraded (6.220%) 2024-03-24T11:43:56.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:56 smithi145 ceph-mon[67080]: pgmap v76: 161 pgs: 8 active+undersized+degraded, 17 active+undersized, 136 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 39/627 objects degraded (6.220%) 2024-03-24T11:43:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:56 smithi149 ceph-mon[58107]: pgmap v76: 161 pgs: 8 active+undersized+degraded, 17 active+undersized, 136 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 39/627 objects degraded (6.220%) 2024-03-24T11:43:57.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:43:57 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:43:57.475+0000 7f4ffb2c5700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:43:58.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[64662]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:43:58.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:43:58.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-24T11:43:58.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:58 smithi145 ceph-mon[67080]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:43:58.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:43:58.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:43:58.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-24T11:43:58.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-mon[58107]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:43:58.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:43:58.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-24T11:43:58.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:43:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:43:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:43:59.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:59 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/627 objects degraded (6.220%), 8 pgs degraded) 2024-03-24T11:43:59.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:43:59 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:43:59.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:59 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/627 objects degraded (6.220%), 8 pgs degraded) 2024-03-24T11:43:59.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:43:59 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:43:59.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:59 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/627 objects degraded (6.220%), 8 pgs degraded) 2024-03-24T11:43:59.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:43:59 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:44:00.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:00 smithi145 ceph-mon[67080]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-24T11:44:00.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:00 smithi145 ceph-mon[64662]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-24T11:44:00.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:00 smithi149 ceph-mon[58107]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-24T11:44:01.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:01 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:01.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:01 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:01.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:01 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:02.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:02 smithi145 ceph-mon[67080]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:02 smithi145 ceph-mon[64662]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:02 smithi149 ceph-mon[58107]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:04 smithi149 ceph-mon[58107]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:44:05.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:04 smithi145 ceph-mon[64662]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:44:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:04 smithi145 ceph-mon[67080]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:44:05.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:44:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:44:05] "GET /metrics HTTP/1.1" 200 38432 "" "Prometheus/2.33.4" 2024-03-24T11:44:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:05 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:05 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:05 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:05.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:05 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:05.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:05 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:05.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:05 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:05 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:05 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:05 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:06.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:06 smithi149 ceph-mon[58107]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:44:07.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:06 smithi145 ceph-mon[64662]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:44:07.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:06 smithi145 ceph-mon[67080]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:44:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:07 smithi149 ceph-mon[58107]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:08.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:07 smithi145 ceph-mon[64662]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:08.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:07 smithi145 ceph-mon[67080]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:08.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:44:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:44:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:44:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:10 smithi149 ceph-mon[58107]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:11.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:10 smithi145 ceph-mon[64662]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:10 smithi145 ceph-mon[67080]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:11 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:12.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:11 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:12.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:11 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:12 smithi149 ceph-mon[58107]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:12 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:44:13.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:12 smithi145 ceph-mon[64662]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:13.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:12 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:44:13.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:12 smithi145 ceph-mon[67080]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:13.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:12 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:44:13.931 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:13 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:44:13.931 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:13 smithi145 ceph-mon[64662]: Upgrade: osd.2 is safe to restart 2024-03-24T11:44:13.931 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:13 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:44:13.931 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:13 smithi145 ceph-mon[67080]: Upgrade: osd.2 is safe to restart 2024-03-24T11:44:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:13 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-24T11:44:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:13 smithi149 ceph-mon[58107]: Upgrade: osd.2 is safe to restart 2024-03-24T11:44:14.816 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[64662]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:14.816 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:14.817 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:44:14.817 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:14.817 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[67080]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:14.817 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:14.817 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:44:14.817 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:14 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:14 smithi149 ceph-mon[58107]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:14 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:14 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:44:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:14 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:15.788 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:15 smithi145 ceph-mon[64662]: Upgrade: Updating osd.2 2024-03-24T11:44:15.788 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:15 smithi145 ceph-mon[64662]: Deploying daemon osd.2 on smithi145 2024-03-24T11:44:15.788 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:44:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:44:15] "GET /metrics HTTP/1.1" 200 38432 "" "Prometheus/2.33.4" 2024-03-24T11:44:15.789 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:15 smithi145 ceph-mon[67080]: Upgrade: Updating osd.2 2024-03-24T11:44:15.789 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:15 smithi145 ceph-mon[67080]: Deploying daemon osd.2 on smithi145 2024-03-24T11:44:15.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:15 smithi149 ceph-mon[58107]: Upgrade: Updating osd.2 2024-03-24T11:44:15.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:15 smithi149 ceph-mon[58107]: Deploying daemon osd.2 on smithi145 2024-03-24T11:44:16.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:16 smithi145 ceph-mon[67080]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:16.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:16 smithi145 ceph-mon[64662]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:16.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:16 smithi149 ceph-mon[58107]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:17.597 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:44:17.617 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:17 smithi145 systemd[1]: Stopping Ceph osd.2 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:44:17.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:17 smithi149 ceph-mon[58107]: osd.2 marked itself down and dead 2024-03-24T11:44:18.074 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:17 smithi145 ceph-mon[67080]: osd.2 marked itself down and dead 2024-03-24T11:44:18.074 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:17 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[38148]: 2024-03-24T11:44:17.616+0000 7fd15cf37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:44:18.074 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:17 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[38148]: 2024-03-24T11:44:17.616+0000 7fd15cf37700 -1 osd.2 116 *** Got signal Terminated *** 2024-03-24T11:44:18.075 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:17 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[38148]: 2024-03-24T11:44:17.616+0000 7fd15cf37700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:44:18.075 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:17 smithi145 ceph-mon[64662]: osd.2 marked itself down and dead 2024-03-24T11:44:18.267 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:44:18.267 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (4m) 25s ago 9m 26.2M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:44:18.267 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (3m) 108s ago 8m 45.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (3m) 25s ago 8m 54.9M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (2m) 108s ago 13m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (3m) 25s ago 15m 500M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (2m) 25s ago 15m 48.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (119s) 108s ago 13m 19.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (2m) 25s ago 13m 40.5M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (4m) 25s ago 9m 17.3M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (3m) 108s ago 9m 19.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (65s) 25s ago 13m 49.4M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (30s) 25s ago 12m 12.7M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (12m) 25s ago 12m 51.3M 2736M 17.2.0 e1d6a67b021e af6ad30c7fb4 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (12m) 25s ago 12m 54.0M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (11m) 108s ago 11m 55.7M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (11m) 108s ago 11m 50.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:44:18.268 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (10m) 108s ago 10m 48.7M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:44:18.269 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (10m) 108s ago 10m 52.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:44:18.269 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (2m) 108s ago 9m 57.8M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:44:18.269 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (8m) 25s ago 8m 71.7M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:44:18.269 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (8m) 108s ago 8m 74.9M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:44:18.269 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (8m) 25s ago 8m 72.4M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:44:18.269 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (8m) 108s ago 8m 73.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:44:18.361 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:18 smithi145 podman[74966]: 2024-03-24 11:44:18.074314904 +0000 UTC m=+0.563901801 container died af6ad30c7fb42478b5c7a8ad9d3fb033dd99eb9650f9e36f7a0d9eff513ec0dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, name=centos-stream, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., ceph=True, io.buildah.version=1.19.8, distribution-scope=public, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870) 2024-03-24T11:44:18.658 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:18 smithi145 podman[74966]: 2024-03-24 11:44:18.474409065 +0000 UTC m=+0.963995959 container cleanup af6ad30c7fb42478b5c7a8ad9d3fb033dd99eb9650f9e36f7a0d9eff513ec0dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, architecture=x86_64, ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.buildah.version=1.19.8, version=8, RELEASE=HEAD, com.redhat.component=centos-stream-container) 2024-03-24T11:44:18.659 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:18 smithi145 bash[74966]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:44:18.711 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:44:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:44:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:44:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-24T11:44:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 7 2024-03-24T11:44:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:44:18.713 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:44:18.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:44:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:44:18.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:18.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:18.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:44:18.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: failed to encode map e117 with expected crc 2024-03-24T11:44:18.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: failed to encode map e117 with expected crc 2024-03-24T11:44:18.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: osdmap e117: 8 total, 7 up, 8 in 2024-03-24T11:44:18.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: failed to encode map e117 with expected crc 2024-03-24T11:44:18.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:18 smithi149 ceph-mon[58107]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.072 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:18 smithi145 podman[75032]: 2024-03-24 11:44:18.742600052 +0000 UTC m=+0.662839061 container remove af6ad30c7fb42478b5c7a8ad9d3fb033dd99eb9650f9e36f7a0d9eff513ec0dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , ceph=True, version=8, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container) 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "7/23 daemons upgraded", 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:44:19.073 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: osdmap e117: 8 total, 7 up, 8 in 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[64662]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:19.074 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:19.075 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:44:19.075 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.075 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.075 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: osdmap e117: 8 total, 7 up, 8 in 2024-03-24T11:44:19.075 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.075 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:18 smithi145 ceph-mon[67080]: failed to encode map e117 with expected crc 2024-03-24T11:44:19.347 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:19 smithi145 podman[75165]: 2024-03-24 11:44:19.02118248 +0000 UTC m=+0.022313339 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:19.347 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:19 smithi145 podman[75165]: 2024-03-24 11:44:19.135338324 +0000 UTC m=+0.136469158 container create ff734172c5877d7a1c8588507050a57e06ca6d128b37833f71a5415f01d29ff1 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T11:44:19.476 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_WARN 1 osds down 2024-03-24T11:44:19.476 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-24T11:44:19.476 INFO:teuthology.orchestra.run.smithi145.stdout: osd.2 (root=default,host=smithi145) is down 2024-03-24T11:44:19.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:19 smithi145 podman[75165]: 2024-03-24 11:44:19.346885468 +0000 UTC m=+0.348016295 container init ff734172c5877d7a1c8588507050a57e06ca6d128b37833f71a5415f01d29ff1 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T11:44:19.612 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:19 smithi145 podman[75165]: 2024-03-24 11:44:19.350505728 +0000 UTC m=+0.351636560 container start ff734172c5877d7a1c8588507050a57e06ca6d128b37833f71a5415f01d29ff1 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:44:19.612 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:19 smithi145 podman[75165]: 2024-03-24 11:44:19.421516962 +0000 UTC m=+0.422647808 container attach ff734172c5877d7a1c8588507050a57e06ca6d128b37833f71a5415f01d29ff1 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:44:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: osdmap e118: 8 total, 7 up, 8 in 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/1929330136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/4172440535' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:20.111 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:19 smithi145 podman[75165]: 2024-03-24 11:44:19.65232995 +0000 UTC m=+0.653460780 container died ff734172c5877d7a1c8588507050a57e06ca6d128b37833f71a5415f01d29ff1 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: osdmap e118: 8 total, 7 up, 8 in 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1929330136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/4172440535' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: osdmap e118: 8 total, 7 up, 8 in 2024-03-24T11:44:20.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/1929330136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:20.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/4172440535' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:44:20.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:20.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:20.539 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 podman[75217]: 2024-03-24 11:44:20.303292264 +0000 UTC m=+0.645747032 container remove ff734172c5877d7a1c8588507050a57e06ca6d128b37833f71a5415f01d29ff1 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-24T11:44:20.539 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-24T11:44:20.539 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 systemd[1]: Stopped Ceph osd.2 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:44:20.539 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.2.service: Consumed 6.671s CPU time. 2024-03-24T11:44:20.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[67080]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[67080]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:20.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[67080]: pgmap v90: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:44:20.862 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 systemd[1]: Starting Ceph osd.2 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:44:20.862 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 podman[75323]: 2024-03-24 11:44:20.659488302 +0000 UTC m=+0.018953408 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:20.862 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 podman[75323]: 2024-03-24 11:44:20.760055473 +0000 UTC m=+0.119520616 container create 8481857d182bad64788fdc90449832aace483113d5b79cfb228811905dcc9f66 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:44:20.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[64662]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[64662]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:20.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:20 smithi145 ceph-mon[64662]: pgmap v90: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:44:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:20 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:20 smithi149 ceph-mon[58107]: failed to encode map e118 with expected crc 2024-03-24T11:44:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:20 smithi149 ceph-mon[58107]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:20 smithi149 ceph-mon[58107]: pgmap v90: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:44:21.362 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 podman[75323]: 2024-03-24 11:44:20.962646007 +0000 UTC m=+0.322111103 container init 8481857d182bad64788fdc90449832aace483113d5b79cfb228811905dcc9f66 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:44:21.362 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:20 smithi145 podman[75323]: 2024-03-24 11:44:20.965782392 +0000 UTC m=+0.325247491 container start 8481857d182bad64788fdc90449832aace483113d5b79cfb228811905dcc9f66 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:21.362 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 podman[75323]: 2024-03-24 11:44:21.037607462 +0000 UTC m=+0.397072560 container attach 8481857d182bad64788fdc90449832aace483113d5b79cfb228811905dcc9f66 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T11:44:21.681 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate[75338]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:44:21.681 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate[75338]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:44:21.681 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 bash[75323]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:44:21.681 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 bash[75323]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:44:21.681 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 bash[75323]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:44:21.681 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 bash[75323]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 bash[75323]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 bash[75323]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 bash[75323]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate[75338]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate[75338]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate[75338]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate[75338]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate[75338]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-24T11:44:21.682 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 podman[75323]: 2024-03-24 11:44:21.434303131 +0000 UTC m=+0.793768219 container died 8481857d182bad64788fdc90449832aace483113d5b79cfb228811905dcc9f66 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T11:44:21.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:21 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:21.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:21 smithi149 ceph-mon[58107]: pgmap v91: 161 pgs: 8 active+undersized+degraded, 4 stale+active+clean, 26 active+undersized, 123 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 29/627 objects degraded (4.625%) 2024-03-24T11:44:21.992 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:21.992 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-mon[67080]: pgmap v91: 161 pgs: 8 active+undersized+degraded, 4 stale+active+clean, 26 active+undersized, 123 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 29/627 objects degraded (4.625%) 2024-03-24T11:44:21.992 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:21.992 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:21 smithi145 ceph-mon[64662]: pgmap v91: 161 pgs: 8 active+undersized+degraded, 4 stale+active+clean, 26 active+undersized, 123 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 29/627 objects degraded (4.625%) 2024-03-24T11:44:22.318 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:21 smithi145 podman[75465]: 2024-03-24 11:44:21.991144343 +0000 UTC m=+0.551369749 container remove 8481857d182bad64788fdc90449832aace483113d5b79cfb228811905dcc9f66 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-activate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-24T11:44:22.319 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 podman[75532]: 2024-03-24 11:44:22.192095787 +0000 UTC m=+0.104000638 container create a91a34aa846516a51e733d3bcaba70625cf16d6fb91c3ea8895bd60a4fc415d4 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:22.319 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 podman[75532]: 2024-03-24 11:44:22.106199529 +0000 UTC m=+0.018104389 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:22.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 podman[75532]: 2024-03-24 11:44:22.411344664 +0000 UTC m=+0.323249524 container init a91a34aa846516a51e733d3bcaba70625cf16d6fb91c3ea8895bd60a4fc415d4 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:44:22.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 podman[75532]: 2024-03-24 11:44:22.414745537 +0000 UTC m=+0.326650392 container start a91a34aa846516a51e733d3bcaba70625cf16d6fb91c3ea8895bd60a4fc415d4 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T11:44:22.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-osd[75550]: -- 172.21.15.145:0/2365712938 <== mon.1 v2:172.21.15.145:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56419a45b1e0 con 0x56419a42d000 2024-03-24T11:44:22.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 bash[75532]: a91a34aa846516a51e733d3bcaba70625cf16d6fb91c3ea8895bd60a4fc415d4 2024-03-24T11:44:22.612 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 systemd[1]: Started Ceph osd.2 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:44:22.959 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:22.960 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:22.960 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:22.960 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:22.960 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:22.960 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:22 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:23.358 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:22 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[75546]: 2024-03-24T11:44:22.958+0000 7fbfb36137c0 -1 Falling back to public interface 2024-03-24T11:44:23.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:23 smithi149 ceph-mon[58107]: pgmap v92: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:24.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:23 smithi145 ceph-mon[64662]: pgmap v92: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:24.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:23 smithi145 ceph-mon[67080]: pgmap v92: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:25.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:44:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:44:25] "GET /metrics HTTP/1.1" 200 38426 "" "Prometheus/2.33.4" 2024-03-24T11:44:25.862 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[75546]: 2024-03-24T11:44:25.676+0000 7fbfb36137c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-24T11:44:25.862 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[75546]: 2024-03-24T11:44:25.696+0000 7fbfb36137c0 -1 osd.2 116 log_to_monitors true 2024-03-24T11:44:26.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:26 smithi149 ceph-mon[58107]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:26.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:26 smithi149 ceph-mon[58107]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:44:27.111 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:44:26 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[75546]: 2024-03-24T11:44:26.667+0000 7fbfa4a6b700 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:44:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:26 smithi145 ceph-mon[64662]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:26 smithi145 ceph-mon[64662]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:44:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:26 smithi145 ceph-mon[67080]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:26 smithi145 ceph-mon[67080]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:44:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:44:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: failed to encode map e119 with expected crc 2024-03-24T11:44:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: osdmap e119: 8 total, 7 up, 8 in 2024-03-24T11:44:27.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:44:27.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:27.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:27.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: failed to encode map e119 with expected crc 2024-03-24T11:44:27.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:27 smithi149 ceph-mon[58107]: failed to encode map e119 with expected crc 2024-03-24T11:44:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:44:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: failed to encode map e119 with expected crc 2024-03-24T11:44:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: osdmap e119: 8 total, 7 up, 8 in 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: failed to encode map e119 with expected crc 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[64662]: failed to encode map e119 with expected crc 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: failed to encode map e119 with expected crc 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: osdmap e119: 8 total, 7 up, 8 in 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: from='osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: failed to encode map e119 with expected crc 2024-03-24T11:44:28.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:27 smithi145 ceph-mon[67080]: failed to encode map e119 with expected crc 2024-03-24T11:44:28.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:44:28.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-24T11:44:28.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:44:28.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: pgmap v95: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244] boot 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:28 smithi149 ceph-mon[58107]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:44:28.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: pgmap v95: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:28.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:44:28.983 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244] boot 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[64662]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: pgmap v95: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.984 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.985 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: osd.2 [v2:172.21.15.145:6818/2587324244,v1:172.21.15.145:6819/2587324244] boot 2024-03-24T11:44:28.985 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:44:28.985 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:44:28.985 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: failed to encode map e120 with expected crc 2024-03-24T11:44:28.985 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.985 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:28.985 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:28 smithi145 ceph-mon[67080]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:44:29.972 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:29 smithi149 ceph-mon[58107]: failed to encode map e121 with expected crc 2024-03-24T11:44:29.973 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:29 smithi149 ceph-mon[58107]: failed to encode map e121 with expected crc 2024-03-24T11:44:29.973 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:29 smithi149 ceph-mon[58107]: failed to encode map e121 with expected crc 2024-03-24T11:44:29.973 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:29 smithi149 ceph-mon[58107]: failed to encode map e121 with expected crc 2024-03-24T11:44:29.973 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:29 smithi149 ceph-mon[58107]: failed to encode map e121 with expected crc 2024-03-24T11:44:29.973 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:29 smithi149 ceph-mon[58107]: failed to encode map e121 with expected crc 2024-03-24T11:44:29.973 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:29 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:30.015 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[64662]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.015 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[64662]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.015 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[64662]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.015 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[64662]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[64662]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[64662]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[67080]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[67080]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[67080]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[67080]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[67080]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[67080]: failed to encode map e121 with expected crc 2024-03-24T11:44:30.016 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:29 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:30 smithi149 ceph-mon[58107]: pgmap v98: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:31.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:30 smithi145 ceph-mon[64662]: pgmap v98: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:31.035 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:30 smithi145 ceph-mon[67080]: pgmap v98: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:44:31.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:44:31 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:44:31.590+0000 7f4ffb2c5700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-24T11:44:32.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:32.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:32.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:32.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: pgmap v99: 161 pgs: 5 active+undersized+degraded, 17 active+undersized, 139 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-03-24T11:44:32.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:32.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:32.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:32.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:32 smithi149 ceph-mon[58107]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-24T11:44:32.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: pgmap v99: 161 pgs: 5 active+undersized+degraded, 17 active+undersized, 139 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:32.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[67080]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: pgmap v99: 161 pgs: 5 active+undersized+degraded, 17 active+undersized, 139 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:32.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:32.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:32.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:32 smithi145 ceph-mon[64662]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-24T11:44:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:33 smithi149 ceph-mon[58107]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-24T11:44:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:33 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded) 2024-03-24T11:44:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:33 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:44:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:33 smithi145 ceph-mon[64662]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-24T11:44:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:33 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded) 2024-03-24T11:44:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:33 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:44:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:33 smithi145 ceph-mon[67080]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-24T11:44:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:33 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded) 2024-03-24T11:44:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:33 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:44:35.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:35 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:35.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:35 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:35.863 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:44:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:44:35] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.33.4" 2024-03-24T11:44:35.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:35 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:35.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:35 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:35.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:35 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:35.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:35 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:36.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:36 smithi145 ceph-mon[64662]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:36.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:36 smithi145 ceph-mon[67080]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:36 smithi149 ceph-mon[58107]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:38 smithi149 ceph-mon[58107]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:44:38.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:44:38 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:44:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:44:39.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:38 smithi145 ceph-mon[64662]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:44:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:38 smithi145 ceph-mon[67080]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:44:40.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:40 smithi149 ceph-mon[58107]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-24T11:44:40.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:40 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:41.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:40 smithi145 ceph-mon[67080]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-24T11:44:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:40 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:40 smithi145 ceph-mon[64662]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-24T11:44:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:40 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:42.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:42 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:42.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:42 smithi149 ceph-mon[58107]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:43.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:42 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:43.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:42 smithi145 ceph-mon[67080]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:42 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:42 smithi145 ceph-mon[64662]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:44.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:44 smithi149 ceph-mon[58107]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:45.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:44 smithi145 ceph-mon[67080]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:45.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:44 smithi145 ceph-mon[64662]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:45.863 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:44:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:44:45] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.33.4" 2024-03-24T11:44:46.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:46 smithi149 ceph-mon[58107]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:46.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:46 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:47.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:46 smithi145 ceph-mon[67080]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:47.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:46 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:47.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:46 smithi145 ceph-mon[64662]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:47.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:46 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:47.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:47 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:47.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:47 smithi145 ceph-mon[67080]: Upgrade: osd.3 is safe to restart 2024-03-24T11:44:47.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:47 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:47.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:47 smithi145 ceph-mon[64662]: Upgrade: osd.3 is safe to restart 2024-03-24T11:44:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:47 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-24T11:44:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:47 smithi149 ceph-mon[58107]: Upgrade: osd.3 is safe to restart 2024-03-24T11:44:48.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:48 smithi149 ceph-mon[58107]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:48.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:48 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:48.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:48 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:44:48.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:48 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:48.976 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:44:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:44:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[67080]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[64662]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:44:48.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:48 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:44:49.804 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:49 smithi145 ceph-mon[64662]: Upgrade: Updating osd.3 2024-03-24T11:44:49.804 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:49 smithi145 ceph-mon[64662]: Deploying daemon osd.3 on smithi145 2024-03-24T11:44:49.804 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:49.804 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:49 smithi145 ceph-mon[67080]: Upgrade: Updating osd.3 2024-03-24T11:44:49.804 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:49 smithi145 ceph-mon[67080]: Deploying daemon osd.3 on smithi145 2024-03-24T11:44:49.804 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:49.862 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:44:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:49 smithi149 ceph-mon[58107]: Upgrade: Updating osd.3 2024-03-24T11:44:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:49 smithi149 ceph-mon[58107]: Deploying daemon osd.3 on smithi145 2024-03-24T11:44:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:44:50.560 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:44:50.560 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (4m) 23s ago 9m 26.2M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:44:50.560 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (4m) 2m ago 9m 45.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:44:50.560 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (3m) 23s ago 8m 55.1M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:44:50.560 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:44:50.560 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (4m) 23s ago 15m 505M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:44:50.560 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (3m) 23s ago 15m 54.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (2m) 2m ago 14m 19.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (2m) 23s ago 14m 38.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (4m) 23s ago 10m 17.3M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (4m) 2m ago 10m 19.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (98s) 23s ago 13m 51.0M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (62s) 23s ago 13m 50.0M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (28s) 23s ago 13m 12.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (12m) 23s ago 12m 55.8M 2736M 17.2.0 e1d6a67b021e e086f48d69d6 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (12m) 2m ago 12m 55.7M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (11m) 2m ago 11m 50.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (11m) 2m ago 11m 48.7M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (11m) 2m ago 11m 52.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (3m) 2m ago 10m 57.8M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:44:50.561 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (8m) 23s ago 8m 72.0M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:44:50.562 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (8m) 2m ago 8m 74.9M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:44:50.562 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (8m) 23s ago 8m 72.6M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:44:50.562 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (8m) 2m ago 8m 73.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:44:50.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:50 smithi145 ceph-mon[64662]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:50.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:50 smithi145 ceph-mon[67080]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:50.963 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:44:50.964 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:44:50.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:44:50.965 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:44:50.965 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-24T11:44:50.965 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:44:50.965 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:44:50.965 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:44:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:50 smithi149 ceph-mon[58107]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:44:51.298 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:44:51.299 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "8/23 daemons upgraded", 2024-03-24T11:44:51.299 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:44:51.299 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:44:51.299 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:44:51.711 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:44:51.943 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:51 smithi145 systemd[1]: Stopping Ceph osd.3 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:44:51.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[67080]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[67080]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[67080]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/1137701070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:51.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[64662]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[64662]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[64662]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1137701070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:51.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:51 smithi149 ceph-mon[58107]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:51 smithi149 ceph-mon[58107]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:51 smithi149 ceph-mon[58107]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:51.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:51 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/1137701070' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:44:52.361 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[41123]: 2024-03-24T11:44:51.941+0000 7fcaf0aa7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:44:52.361 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[41123]: 2024-03-24T11:44:51.941+0000 7fcaf0aa7700 -1 osd.3 121 *** Got signal Terminated *** 2024-03-24T11:44:52.361 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[41123]: 2024-03-24T11:44:51.941+0000 7fcaf0aa7700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:44:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:52 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:52 smithi149 ceph-mon[58107]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:52 smithi149 ceph-mon[58107]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:52 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/675586517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:44:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:52 smithi149 ceph-mon[58107]: osd.3 marked itself down and dead 2024-03-24T11:44:53.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[64662]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[64662]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/675586517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[64662]: osd.3 marked itself down and dead 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[67080]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[67080]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/675586517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:44:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:52 smithi145 ceph-mon[67080]: osd.3 marked itself down and dead 2024-03-24T11:44:53.443 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:53 smithi145 podman[77541]: 2024-03-24 11:44:53.162578001 +0000 UTC m=+1.331053420 container died e086f48d69d6f65c9783eaa7cb2138338be2fd0984770d8f83ab62c727aeddf3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, vendor=Red Hat, Inc., io.buildah.version=1.19.8, architecture=x86_64, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, RELEASE=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , distribution-scope=public, version=8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-24T11:44:53.818 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:44:53.818 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.818 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.818 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: osdmap e122: 8 total, 7 up, 8 in 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[64662]: pgmap v111: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: osdmap e122: 8 total, 7 up, 8 in 2024-03-24T11:44:53.819 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.820 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.820 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.820 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.820 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:53 smithi145 ceph-mon[67080]: pgmap v111: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:44:53.820 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:53 smithi145 podman[77541]: 2024-03-24 11:44:53.550050462 +0000 UTC m=+1.718525879 container cleanup e086f48d69d6f65c9783eaa7cb2138338be2fd0984770d8f83ab62c727aeddf3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0) 2024-03-24T11:44:53.820 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:53 smithi145 bash[77541]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3 2024-03-24T11:44:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:44:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: osdmap e122: 8 total, 7 up, 8 in 2024-03-24T11:44:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: failed to encode map e122 with expected crc 2024-03-24T11:44:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:53 smithi149 ceph-mon[58107]: pgmap v111: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:44:54.111 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:53 smithi145 podman[77569]: 2024-03-24 11:44:53.817967431 +0000 UTC m=+0.651755560 container remove e086f48d69d6f65c9783eaa7cb2138338be2fd0984770d8f83ab62c727aeddf3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vendor=Red Hat, Inc., distribution-scope=public, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-24T11:44:54.496 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:54 smithi145 podman[77662]: 2024-03-24 11:44:54.062812839 +0000 UTC m=+0.019003332 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:54.496 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:54 smithi145 podman[77662]: 2024-03-24 11:44:54.177302506 +0000 UTC m=+0.133492996 container create a56f992b4dc164ff4512f846f7ca8989cb42bc33d8344002d20d9b0f35959ff8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T11:44:54.496 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:54 smithi145 podman[77662]: 2024-03-24 11:44:54.396727491 +0000 UTC m=+0.352917980 container init a56f992b4dc164ff4512f846f7ca8989cb42bc33d8344002d20d9b0f35959ff8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-24T11:44:54.497 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:54 smithi145 podman[77662]: 2024-03-24 11:44:54.399883421 +0000 UTC m=+0.356073919 container start a56f992b4dc164ff4512f846f7ca8989cb42bc33d8344002d20d9b0f35959ff8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b) 2024-03-24T11:44:54.497 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:54 smithi145 podman[77662]: 2024-03-24 11:44:54.496436367 +0000 UTC m=+0.452626872 container attach a56f992b4dc164ff4512f846f7ca8989cb42bc33d8344002d20d9b0f35959ff8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:44:54.861 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:54 smithi145 podman[77662]: 2024-03-24 11:44:54.694528664 +0000 UTC m=+0.650719163 container died a56f992b4dc164ff4512f846f7ca8989cb42bc33d8344002d20d9b0f35959ff8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: osdmap e123: 8 total, 7 up, 8 in 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[64662]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: osdmap e123: 8 total, 7 up, 8 in 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:54 smithi145 ceph-mon[67080]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: osdmap e123: 8 total, 7 up, 8 in 2024-03-24T11:44:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:54.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:54 smithi149 ceph-mon[58107]: failed to encode map e123 with expected crc 2024-03-24T11:44:55.527 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 podman[77689]: 2024-03-24 11:44:55.25858383 +0000 UTC m=+0.558083076 container remove a56f992b4dc164ff4512f846f7ca8989cb42bc33d8344002d20d9b0f35959ff8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:44:55.527 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-24T11:44:55.528 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 systemd[1]: Stopped Ceph osd.3 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:44:55.528 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.3.service: Consumed 8.470s CPU time. 2024-03-24T11:44:55.528 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 systemd[1]: Starting Ceph osd.3 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:44:55.823 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:55 smithi145 ceph-mon[67080]: pgmap v113: 161 pgs: 14 active+undersized+degraded, 13 stale+active+clean, 18 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 41/627 objects degraded (6.539%) 2024-03-24T11:44:55.823 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:44:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:44:55] "GET /metrics HTTP/1.1" 200 38445 "" "Prometheus/2.33.4" 2024-03-24T11:44:55.824 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 podman[77798]: 2024-03-24 11:44:55.605238392 +0000 UTC m=+0.019138502 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:55.824 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 podman[77798]: 2024-03-24 11:44:55.717803806 +0000 UTC m=+0.131703907 container create 82b35d93a8c1a3bcda9db7601f1c4c82737f0474925b93b24acd8eeba7b428e5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T11:44:55.824 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:55 smithi145 ceph-mon[64662]: pgmap v113: 161 pgs: 14 active+undersized+degraded, 13 stale+active+clean, 18 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 41/627 objects degraded (6.539%) 2024-03-24T11:44:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:55 smithi149 ceph-mon[58107]: pgmap v113: 161 pgs: 14 active+undersized+degraded, 13 stale+active+clean, 18 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 41/627 objects degraded (6.539%) 2024-03-24T11:44:56.111 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 podman[77798]: 2024-03-24 11:44:55.920350849 +0000 UTC m=+0.334250958 container init 82b35d93a8c1a3bcda9db7601f1c4c82737f0474925b93b24acd8eeba7b428e5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-24T11:44:56.111 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:55 smithi145 podman[77798]: 2024-03-24 11:44:55.923564057 +0000 UTC m=+0.337464163 container start 82b35d93a8c1a3bcda9db7601f1c4c82737f0474925b93b24acd8eeba7b428e5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T11:44:56.111 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 podman[77798]: 2024-03-24 11:44:56.003547039 +0000 UTC m=+0.417447146 container attach 82b35d93a8c1a3bcda9db7601f1c4c82737f0474925b93b24acd8eeba7b428e5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-db0330b) 2024-03-24T11:44:56.691 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate[77813]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 bash[77798]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 bash[77798]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 bash[77798]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 bash[77798]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 bash[77798]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 bash[77798]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 bash[77798]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate[77813]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate[77813]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate[77813]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate[77813]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate[77813]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate[77813]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-24T11:44:56.692 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:56 smithi145 podman[77798]: 2024-03-24 11:44:56.38981913 +0000 UTC m=+0.803719231 container died 82b35d93a8c1a3bcda9db7601f1c4c82737f0474925b93b24acd8eeba7b428e5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0) 2024-03-24T11:44:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:56 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 41/627 objects degraded (6.539%), 14 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:57.050 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 41/627 objects degraded (6.539%), 14 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:57.050 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:56 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 41/627 objects degraded (6.539%), 14 pgs degraded (PG_DEGRADED) 2024-03-24T11:44:57.361 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 podman[77935]: 2024-03-24 11:44:57.050280165 +0000 UTC m=+0.655227426 container remove 82b35d93a8c1a3bcda9db7601f1c4c82737f0474925b93b24acd8eeba7b428e5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T11:44:57.361 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 podman[77998]: 2024-03-24 11:44:57.169995905 +0000 UTC m=+0.018541941 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:57.362 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 podman[77998]: 2024-03-24 11:44:57.276129492 +0000 UTC m=+0.124675560 container create ffbb9bef1b39b18e1035b8f8d2360692f0b66eb9b7ceea3a93f5f357c0703ab9 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T11:44:57.862 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 podman[77998]: 2024-03-24 11:44:57.486817128 +0000 UTC m=+0.335363171 container init ffbb9bef1b39b18e1035b8f8d2360692f0b66eb9b7ceea3a93f5f357c0703ab9 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5) 2024-03-24T11:44:57.862 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 podman[77998]: 2024-03-24 11:44:57.490190856 +0000 UTC m=+0.338736879 container start ffbb9bef1b39b18e1035b8f8d2360692f0b66eb9b7ceea3a93f5f357c0703ab9 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T11:44:57.862 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 ceph-osd[78017]: -- 172.21.15.145:0/3122783888 <== mon.1 v2:172.21.15.145:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56125fc491e0 con 0x56125fc1b400 2024-03-24T11:44:57.862 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 bash[77998]: ffbb9bef1b39b18e1035b8f8d2360692f0b66eb9b7ceea3a93f5f357c0703ab9 2024-03-24T11:44:57.862 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:57 smithi145 systemd[1]: Started Ceph osd.3 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:44:58.362 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:44:58 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[78013]: 2024-03-24T11:44:58.030+0000 7fb6ccf967c0 -1 Falling back to public interface 2024-03-24T11:44:58.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:58 smithi145 ceph-mon[64662]: pgmap v114: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:44:58.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:58.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:44:58 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:58.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:58 smithi145 ceph-mon[67080]: pgmap v114: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:44:58.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:58.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:44:58 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:58 smithi149 ceph-mon[58107]: pgmap v114: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:44:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:44:58 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:44:58.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:44:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:44:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:45:00.942 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:00 smithi145 ceph-mon[64662]: pgmap v115: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:00.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:00 smithi145 ceph-mon[67080]: pgmap v115: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:00.943 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:45:00 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[78013]: 2024-03-24T11:45:00.753+0000 7fb6ccf967c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-24T11:45:00.943 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:45:00 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[78013]: 2024-03-24T11:45:00.781+0000 7fb6ccf967c0 -1 osd.3 121 log_to_monitors true 2024-03-24T11:45:00.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:00 smithi149 ceph-mon[58107]: pgmap v115: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:01.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:01 smithi145 ceph-mon[67080]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:45:01.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:01 smithi145 ceph-mon[64662]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:45:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:01 smithi149 ceph-mon[58107]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:45:02.611 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[78013]: 2024-03-24T11:45:02.160+0000 7fb6ba3e6700 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:45:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:45:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: failed to encode map e124 with expected crc 2024-03-24T11:45:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: osdmap e124: 8 total, 7 up, 8 in 2024-03-24T11:45:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:45:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: failed to encode map e124 with expected crc 2024-03-24T11:45:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' 2024-03-24T11:45:02.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:02 smithi149 ceph-mon[58107]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: osdmap e124: 8 total, 7 up, 8 in 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' 2024-03-24T11:45:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[64662]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: pgmap v116: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: osdmap e124: 8 total, 7 up, 8 in 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: from='osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676]' entity='osd.3' 2024-03-24T11:45:03.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:02 smithi145 ceph-mon[67080]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:45:03.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.943 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676] boot 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676] boot 2024-03-24T11:45:03.944 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:45:03.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:45:03.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:03 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: failed to encode map e124 with expected crc 2024-03-24T11:45:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:45:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: osd.3 [v2:172.21.15.145:6826/1177811676,v1:172.21.15.145:6827/1177811676] boot 2024-03-24T11:45:03.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:45:03.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:45:03.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: failed to encode map e125 with expected crc 2024-03-24T11:45:03.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:03.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:03 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e125 with expected crc 2024-03-24T11:45:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: pgmap v119: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:45:04.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: failed to encode map e126 with expected crc 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:04.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:05.012 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e125 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: pgmap v119: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.013 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e125 with expected crc 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: pgmap v119: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.014 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: failed to encode map e126 with expected crc 2024-03-24T11:45:05.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:05.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:05.015 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:05.612 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:45:05] "GET /metrics HTTP/1.1" 200 38309 "" "Prometheus/2.33.4" 2024-03-24T11:45:05.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:05 smithi145 ceph-mon[64662]: pgmap v121: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-24T11:45:05.944 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:05 smithi145 ceph-mon[67080]: pgmap v121: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-24T11:45:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:05 smithi149 ceph-mon[58107]: pgmap v121: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-24T11:45:06.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:06 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:45:06.829+0000 7f4ffb2c5700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:45:07.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:07.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:07.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:08 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:08 smithi149 ceph-mon[58107]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:45:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:08 smithi149 ceph-mon[58107]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:08 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/627 objects degraded (9.729%), 16 pgs degraded) 2024-03-24T11:45:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:08 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:45:08.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:45:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:45:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"smithi145:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.145\", device_class=\"ssd\", hostname=\"smithi145\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.145\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-24T11:45:09.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[64662]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[64662]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/627 objects degraded (9.729%), 16 pgs degraded) 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[67080]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[67080]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/627 objects degraded (9.729%), 16 pgs degraded) 2024-03-24T11:45:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:08 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:45:10.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:09 smithi145 ceph-mon[64662]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:10.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:09 smithi145 ceph-mon[67080]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:09 smithi149 ceph-mon[58107]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:12 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:12 smithi149 ceph-mon[58107]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:13.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:12 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:12 smithi145 ceph-mon[67080]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:13.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:12 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:13.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:12 smithi145 ceph-mon[64662]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:45:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:14 smithi149 ceph-mon[58107]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-24T11:45:15.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:14 smithi145 ceph-mon[67080]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-24T11:45:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:14 smithi145 ceph-mon[64662]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-24T11:45:15.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:45:15] "GET /metrics HTTP/1.1" 200 38309 "" "Prometheus/2.33.4" 2024-03-24T11:45:15.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:15 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:15.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:15 smithi145 ceph-mon[67080]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2024-03-24T11:45:15.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:15 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:15.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:15 smithi145 ceph-mon[64662]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2024-03-24T11:45:15.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:15 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:15.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:15 smithi149 ceph-mon[58107]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2024-03-24T11:45:18.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:18 smithi149 ceph-mon[58107]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:45:19.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:18 smithi145 ceph-mon[67080]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:45:19.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:18 smithi145 ceph-mon[64662]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:45:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:20 smithi149 ceph-mon[58107]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:21.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:20 smithi145 ceph-mon[67080]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:20 smithi145 ceph-mon[64662]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:22.070 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:45:22.715 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:45:22.715 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (5m) 20s ago 10m 25.6M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (4m) 2m ago 9m 45.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (4m) 20s ago 9m 55.5M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (4m) 20s ago 16m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (3m) 20s ago 16m 49.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (3m) 2m ago 14m 19.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (3m) 20s ago 15m 40.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (5m) 20s ago 10m 17.5M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (4m) 2m ago 10m 19.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (2m) 20s ago 14m 51.2M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (94s) 20s ago 13m 51.7M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (60s) 20s ago 13m 41.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:45:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (25s) 20s ago 13m 12.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (12m) 2m ago 12m 55.7M 1454M 17.2.0 e1d6a67b021e 02b6c464b6e4 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (12m) 2m ago 12m 50.3M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (12m) 2m ago 12m 48.7M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (11m) 2m ago 11m 52.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (4m) 2m ago 10m 57.8M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (9m) 20s ago 9m 72.2M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (9m) 2m ago 9m 74.9M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (9m) 20s ago 9m 72.8M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:45:22.717 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (9m) 2m ago 9m 73.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:22 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:22 smithi149 ceph-mon[58107]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:22.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:23.108 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:45:23.108 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:45:23.108 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 9 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:45:23.109 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:45:23.110 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:22 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:23.110 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:22 smithi145 ceph-mon[67080]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:23.110 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:23.110 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:22 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:23.110 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:22 smithi145 ceph-mon[64662]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:23.110 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:23.438 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:45:23.438 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:45:23.438 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:45:23.438 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:45:23.438 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:45:23.438 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:45:23.439 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:45:23.439 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:45:23.439 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "9/23 daemons upgraded", 2024-03-24T11:45:23.439 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:45:23.439 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:45:23.439 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:45:23.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:23.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: Upgrade: osd.4 is safe to restart 2024-03-24T11:45:23.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:23.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:23.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2041739830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:23.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:23.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:45:23.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:23 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:23.829 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:45:23.829 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:23.829 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: Upgrade: osd.4 is safe to restart 2024-03-24T11:45:23.829 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:23.829 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:23.829 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2041739830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: Upgrade: osd.4 is safe to restart 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2041739830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:45:23.830 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:23 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:24 smithi149 ceph-mon[58107]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:24 smithi149 ceph-mon[58107]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:24 smithi149 ceph-mon[58107]: Upgrade: Updating osd.4 2024-03-24T11:45:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:24 smithi149 ceph-mon[58107]: Deploying daemon osd.4 on smithi149 2024-03-24T11:45:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:24 smithi149 ceph-mon[58107]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:24.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:24 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2057979072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:45:25.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[64662]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:25.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[64662]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[64662]: Upgrade: Updating osd.4 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[64662]: Deploying daemon osd.4 on smithi149 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[64662]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2057979072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[67080]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[67080]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[67080]: Upgrade: Updating osd.4 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[67080]: Deploying daemon osd.4 on smithi149 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[67080]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:24 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2057979072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:45:25.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:45:25] "GET /metrics HTTP/1.1" 200 38458 "" "Prometheus/2.33.4" 2024-03-24T11:45:26.904 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:26 smithi149 ceph-mon[58107]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:27.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:26 smithi145 ceph-mon[67080]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:26 smithi145 ceph-mon[64662]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:27.197 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:26 smithi149 systemd[1]: Stopping Ceph osd.4 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:45:27.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[29136]: 2024-03-24T11:45:27.196+0000 7f9a46582700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:45:27.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[29136]: 2024-03-24T11:45:27.196+0000 7f9a46582700 -1 osd.4 126 *** Got signal Terminated *** 2024-03-24T11:45:27.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:27 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[29136]: 2024-03-24T11:45:27.196+0000 7f9a46582700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:45:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:27 smithi149 ceph-mon[58107]: osd.4 marked itself down and dead 2024-03-24T11:45:28.004 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:27 smithi145 ceph-mon[67080]: osd.4 marked itself down and dead 2024-03-24T11:45:28.004 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:27 smithi145 ceph-mon[64662]: osd.4 marked itself down and dead 2024-03-24T11:45:28.472 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:28 smithi149 podman[60909]: 2024-03-24 11:45:28.205747143 +0000 UTC m=+1.151741462 container died 02b6c464b6e471178245e0d5b5541f5a8def13b6879b5913deed8a661170cc31 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, io.buildah.version=1.19.8, vendor=Red Hat, Inc., architecture=x86_64, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , version=8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=) 2024-03-24T11:45:28.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:28 smithi149 ceph-mon[58107]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:28.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:28 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:45:28.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:28 smithi149 ceph-mon[58107]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:28 smithi149 ceph-mon[58107]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:28 smithi149 ceph-mon[58107]: osdmap e127: 8 total, 7 up, 8 in 2024-03-24T11:45:28.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:28 smithi149 ceph-mon[58107]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:28 smithi149 ceph-mon[58107]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.726 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:28 smithi149 podman[60909]: 2024-03-24 11:45:28.571343538 +0000 UTC m=+1.517337926 container cleanup 02b6c464b6e471178245e0d5b5541f5a8def13b6879b5913deed8a661170cc31 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_CLEAN=True, version=8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, distribution-scope=public, io.buildah.version=1.19.8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-24T11:45:28.726 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:28 smithi149 bash[60909]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4 2024-03-24T11:45:28.933 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[67080]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:28.933 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:45:28.933 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[67080]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.933 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[67080]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.933 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[67080]: osdmap e127: 8 total, 7 up, 8 in 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[67080]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[67080]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[64662]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[64662]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[64662]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[64662]: osdmap e127: 8 total, 7 up, 8 in 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[64662]: failed to encode map e127 with expected crc 2024-03-24T11:45:28.934 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:28 smithi145 ceph-mon[64662]: failed to encode map e127 with expected crc 2024-03-24T11:45:29.194 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:28 smithi149 podman[60937]: 2024-03-24 11:45:28.856024846 +0000 UTC m=+0.652901257 container remove 02b6c464b6e471178245e0d5b5541f5a8def13b6879b5913deed8a661170cc31 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-24T11:45:29.451 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:29 smithi149 podman[61030]: 2024-03-24 11:45:29.096474991 +0000 UTC m=+0.018680597 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:29.451 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:29 smithi149 podman[61030]: 2024-03-24 11:45:29.232306039 +0000 UTC m=+0.154511664 container create fa9ee3b3c4ab3b0337220c41f31decfe6c75e087271ec25994a6231f09ca415d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T11:45:29.452 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:29 smithi149 podman[61030]: 2024-03-24 11:45:29.451401805 +0000 UTC m=+0.373607398 container init fa9ee3b3c4ab3b0337220c41f31decfe6c75e087271ec25994a6231f09ca415d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-24T11:45:29.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: osdmap e128: 8 total, 7 up, 8 in 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:29 smithi149 ceph-mon[58107]: failed to encode map e128 with expected crc 2024-03-24T11:45:29.726 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:29 smithi149 podman[61030]: 2024-03-24 11:45:29.454630076 +0000 UTC m=+0.376835664 container start fa9ee3b3c4ab3b0337220c41f31decfe6c75e087271ec25994a6231f09ca415d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:45:29.727 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:29 smithi149 podman[61030]: 2024-03-24 11:45:29.534746636 +0000 UTC m=+0.456952244 container attach fa9ee3b3c4ab3b0337220c41f31decfe6c75e087271ec25994a6231f09ca415d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T11:45:30.038 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:29 smithi149 podman[61030]: 2024-03-24 11:45:29.746938708 +0000 UTC m=+0.669144292 container died fa9ee3b3c4ab3b0337220c41f31decfe6c75e087271ec25994a6231f09ca415d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:45:30.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: osdmap e128: 8 total, 7 up, 8 in 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[67080]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: osdmap e128: 8 total, 7 up, 8 in 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:29 smithi145 ceph-mon[64662]: failed to encode map e128 with expected crc 2024-03-24T11:45:30.670 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:30 smithi149 podman[61057]: 2024-03-24 11:45:30.338483999 +0000 UTC m=+0.585487025 container remove fa9ee3b3c4ab3b0337220c41f31decfe6c75e087271ec25994a6231f09ca415d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T11:45:30.670 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:30 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-24T11:45:30.670 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:30 smithi149 systemd[1]: Stopped Ceph osd.4 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:45:30.670 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:30 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.4.service: Consumed 9.208s CPU time. 2024-03-24T11:45:30.670 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:30 smithi149 systemd[1]: Starting Ceph osd.4 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:45:30.933 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:30 smithi149 ceph-mon[58107]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:30.933 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:30 smithi149 podman[61161]: 2024-03-24 11:45:30.685378951 +0000 UTC m=+0.019643081 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:30.933 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:30 smithi149 podman[61161]: 2024-03-24 11:45:30.806657088 +0000 UTC m=+0.140921210 container create 21b20512476f03d1ecaf762f8d5ebe7192b56a379d3c0f11bc785af0f7485dab (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2) 2024-03-24T11:45:31.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:30 smithi145 ceph-mon[67080]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:30 smithi145 ceph-mon[64662]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:31.456 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 podman[61161]: 2024-03-24 11:45:31.00934676 +0000 UTC m=+0.343610890 container init 21b20512476f03d1ecaf762f8d5ebe7192b56a379d3c0f11bc785af0f7485dab (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T11:45:31.456 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 podman[61161]: 2024-03-24 11:45:31.012509849 +0000 UTC m=+0.346773978 container start 21b20512476f03d1ecaf762f8d5ebe7192b56a379d3c0f11bc785af0f7485dab (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T11:45:31.456 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 podman[61161]: 2024-03-24 11:45:31.100836639 +0000 UTC m=+0.435100780 container attach 21b20512476f03d1ecaf762f8d5ebe7192b56a379d3c0f11bc785af0f7485dab (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T11:45:31.721 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate[61176]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:45:31.721 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate[61176]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:45:31.721 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 bash[61161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:45:31.721 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 bash[61161]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 bash[61161]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 bash[61161]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 bash[61161]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 bash[61161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 bash[61161]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate[61176]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate[61176]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate[61176]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate[61176]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate[61176]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-24T11:45:31.722 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:31 smithi149 podman[61161]: 2024-03-24 11:45:31.480225882 +0000 UTC m=+0.814490003 container died 21b20512476f03d1ecaf762f8d5ebe7192b56a379d3c0f11bc785af0f7485dab (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T11:45:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-mon[58107]: pgmap v136: 161 pgs: 11 active+undersized+degraded, 14 stale+active+clean, 21 active+undersized, 115 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:45:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:31 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 55/627 objects degraded (8.772%), 11 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:32.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:31 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:31 smithi145 ceph-mon[67080]: pgmap v136: 161 pgs: 11 active+undersized+degraded, 14 stale+active+clean, 21 active+undersized, 115 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:45:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:31 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 55/627 objects degraded (8.772%), 11 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:31 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:31 smithi145 ceph-mon[64662]: pgmap v136: 161 pgs: 11 active+undersized+degraded, 14 stale+active+clean, 21 active+undersized, 115 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-03-24T11:45:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:31 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 55/627 objects degraded (8.772%), 11 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:32.453 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 podman[61309]: 2024-03-24 11:45:32.096456865 +0000 UTC m=+0.610598819 container remove 21b20512476f03d1ecaf762f8d5ebe7192b56a379d3c0f11bc785af0f7485dab (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T11:45:32.453 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 podman[61375]: 2024-03-24 11:45:32.213387074 +0000 UTC m=+0.018041816 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:32.453 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 podman[61375]: 2024-03-24 11:45:32.322321693 +0000 UTC m=+0.126976433 container create 8a8a001df0e15bda346dca747effce02cb67fd2eb15eb68d0d63436bbbeb5757 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T11:45:32.725 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 podman[61375]: 2024-03-24 11:45:32.558233891 +0000 UTC m=+0.362888638 container init 8a8a001df0e15bda346dca747effce02cb67fd2eb15eb68d0d63436bbbeb5757 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b) 2024-03-24T11:45:32.725 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 podman[61375]: 2024-03-24 11:45:32.561233158 +0000 UTC m=+0.365887898 container start 8a8a001df0e15bda346dca747effce02cb67fd2eb15eb68d0d63436bbbeb5757 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:45:32.725 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 ceph-osd[61393]: -- 172.21.15.149:0/154932912 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0dedcf1e0 con 0x55d0deda0c00 2024-03-24T11:45:32.725 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 bash[61375]: 8a8a001df0e15bda346dca747effce02cb67fd2eb15eb68d0d63436bbbeb5757 2024-03-24T11:45:32.725 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:32 smithi149 systemd[1]: Started Ceph osd.4 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:45:33.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:33 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:45:33.100+0000 7f6b00b5f7c0 -1 Falling back to public interface 2024-03-24T11:45:33.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:33 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:33.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:33 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:33.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:33 smithi149 ceph-mon[58107]: pgmap v137: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:34.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:33 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:34.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:33 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:33 smithi145 ceph-mon[67080]: pgmap v137: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:33 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:33 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:34.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:33 smithi145 ceph-mon[64662]: pgmap v137: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:35.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:35 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:35.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:35 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:35.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:35 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:35.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:35 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:35.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:45:35] "GET /metrics HTTP/1.1" 200 38460 "" "Prometheus/2.33.4" 2024-03-24T11:45:35.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:35 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:35.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:35 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:36.225 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:35 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:45:35.830+0000 7f6b00b5f7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-24T11:45:36.225 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:35 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:45:35.858+0000 7f6b00b5f7c0 -1 osd.4 126 log_to_monitors true 2024-03-24T11:45:36.909 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:36 smithi149 ceph-mon[58107]: pgmap v138: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:36.909 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:36 smithi149 ceph-mon[58107]: from='osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:45:36.909 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:36 smithi149 ceph-mon[58107]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:45:37.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:36 smithi145 ceph-mon[67080]: pgmap v138: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:37.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:36 smithi145 ceph-mon[67080]: from='osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:45:37.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:36 smithi145 ceph-mon[67080]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:45:37.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:36 smithi145 ceph-mon[64662]: pgmap v138: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 176 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:37.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:36 smithi145 ceph-mon[64662]: from='osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:45:37.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:36 smithi145 ceph-mon[64662]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:45:37.225 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:45:37.120+0000 7f6aedfaf700 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:45:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:45:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: failed to encode map e129 with expected crc 2024-03-24T11:45:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: from='osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:45:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: osdmap e129: 8 total, 7 up, 8 in 2024-03-24T11:45:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:45:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: failed to encode map e129 with expected crc 2024-03-24T11:45:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: from='osd.4 ' entity='osd.4' 2024-03-24T11:45:37.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:37.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:37 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: failed to encode map e129 with expected crc 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: from='osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: osdmap e129: 8 total, 7 up, 8 in 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: failed to encode map e129 with expected crc 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: from='osd.4 ' entity='osd.4' 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: failed to encode map e129 with expected crc 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: from='osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: osdmap e129: 8 total, 7 up, 8 in 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: failed to encode map e129 with expected crc 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: from='osd.4 ' entity='osd.4' 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:37 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: failed to encode map e129 with expected crc 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: pgmap v140: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: failed to encode map e130 with expected crc 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300] boot 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: failed to encode map e129 with expected crc 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: pgmap v140: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: failed to encode map e130 with expected crc 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300] boot 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:38 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: failed to encode map e129 with expected crc 2024-03-24T11:45:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: pgmap v140: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 104/627 objects degraded (16.587%) 2024-03-24T11:45:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:45:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: failed to encode map e130 with expected crc 2024-03-24T11:45:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: osd.4 [v2:172.21.15.149:6800/3965600300,v1:172.21.15.149:6801/3965600300] boot 2024-03-24T11:45:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:45:38.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:45:38.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:38.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:38 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e130 with expected crc 2024-03-24T11:45:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e130 with expected crc 2024-03-24T11:45:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:45:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:39 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 104/627 objects degraded (16.587%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:40.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e130 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e130 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 104/627 objects degraded (16.587%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e130 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e130 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:39 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 104/627 objects degraded (16.587%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T11:45:40.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:40 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:40 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:40 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:40 smithi149 ceph-mon[58107]: failed to encode map e131 with expected crc 2024-03-24T11:45:40.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:40 smithi149 ceph-mon[58107]: pgmap v143: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 104/627 objects degraded (16.587%) 2024-03-24T11:45:41.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[64662]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[64662]: pgmap v143: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 104/627 objects degraded (16.587%) 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[67080]: failed to encode map e131 with expected crc 2024-03-24T11:45:41.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:40 smithi145 ceph-mon[67080]: pgmap v143: 161 pgs: 24 active+undersized+degraded, 44 active+undersized, 93 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 104/627 objects degraded (16.587%) 2024-03-24T11:45:42.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:45:42.189+0000 7f4ffb2c5700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T11:45:42.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:42.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: pgmap v144: 161 pgs: 16 active+undersized+degraded, 29 active+undersized, 116 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-03-24T11:45:42.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:42.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:42.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: pgmap v144: 161 pgs: 16 active+undersized+degraded, 29 active+undersized, 116 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: pgmap v144: 161 pgs: 16 active+undersized+degraded, 29 active+undersized, 116 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:43.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:43.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:43 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:43.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:43 smithi149 ceph-mon[58107]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T11:45:44.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:43 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:44.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:43 smithi145 ceph-mon[64662]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T11:45:44.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:43 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:44.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:43 smithi145 ceph-mon[67080]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-24T11:45:44.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:44 smithi149 ceph-mon[58107]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:45:44.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:44 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/627 objects degraded (8.931%), 16 pgs degraded) 2024-03-24T11:45:44.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:44 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:45:45.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:44 smithi145 ceph-mon[64662]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:45:45.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:44 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/627 objects degraded (8.931%), 16 pgs degraded) 2024-03-24T11:45:45.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:44 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:45:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:44 smithi145 ceph-mon[67080]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:45:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:44 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/627 objects degraded (8.931%), 16 pgs degraded) 2024-03-24T11:45:45.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:44 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:45:45.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:45:45] "GET /metrics HTTP/1.1" 200 38460 "" "Prometheus/2.33.4" 2024-03-24T11:45:46.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:46 smithi149 ceph-mon[58107]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:45:47.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:46 smithi145 ceph-mon[64662]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:45:47.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:46 smithi145 ceph-mon[67080]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:45:48.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:48 smithi149 ceph-mon[58107]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:45:49.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:48 smithi145 ceph-mon[67080]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:45:49.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:48 smithi145 ceph-mon[64662]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:45:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:50 smithi149 ceph-mon[58107]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-24T11:45:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:50.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:50.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:51.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[67080]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-24T11:45:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:51.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[64662]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-24T11:45:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:45:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:45:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:52 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:52 smithi149 ceph-mon[58107]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:53.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:52 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:53.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:52 smithi145 ceph-mon[64662]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:52 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:45:53.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:52 smithi145 ceph-mon[67080]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:54.185 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:45:54.833 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:45:54.833 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (5m) 52s ago 10m 25.6M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:45:54.833 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (5m) 17s ago 10m 52.8M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:45:54.833 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (4m) 52s ago 9m 55.5M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:45:54.833 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (4m) 17s ago 15m 450M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (5m) 52s ago 16m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (4m) 52s ago 16m 49.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (3m) 17s ago 15m 37.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (3m) 52s ago 15m 40.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (5m) 52s ago 11m 17.5M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (5m) 17s ago 11m 18.7M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (2m) 52s ago 14m 51.2M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (2m) 52s ago 14m 51.7M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (92s) 52s ago 14m 41.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (57s) 52s ago 13m 12.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (22s) 17s ago 13m 14.8M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 8a8a001df0e1 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (12m) 17s ago 12m 57.1M 1454M 17.2.0 e1d6a67b021e 8d60e7746b35 2024-03-24T11:45:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (12m) 17s ago 12m 55.5M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:45:54.835 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (12m) 17s ago 12m 58.9M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:45:54.835 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (4m) 17s ago 11m 68.9M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:45:54.835 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (9m) 52s ago 9m 72.2M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:45:54.835 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (9m) 17s ago 9m 75.8M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:45:54.835 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (9m) 52s ago 9m 72.8M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:45:54.835 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (9m) 17s ago 9m 74.9M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:45:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:54 smithi149 ceph-mon[58107]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:55.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:54 smithi145 ceph-mon[64662]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:55.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:54 smithi145 ceph-mon[67080]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 5 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:55.236 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:45:55.237 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:45:55.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:45:55.237 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:45:55.237 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-24T11:45:55.237 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 10 2024-03-24T11:45:55.237 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:45:55.237 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "10/23 daemons upgraded", 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:45:55.564 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:45:55.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:55 smithi145 ceph-mon[67080]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:55.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:55 smithi145 ceph-mon[67080]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:55.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:55 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/4008855635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:55 smithi145 ceph-mon[64662]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:55 smithi145 ceph-mon[64662]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:55 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/4008855635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:55.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:45:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:45:55] "GET /metrics HTTP/1.1" 200 38464 "" "Prometheus/2.33.4" 2024-03-24T11:45:55.954 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:45:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:55 smithi149 ceph-mon[58107]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:55 smithi149 ceph-mon[58107]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:55 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/4008855635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:45:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:56 smithi149 ceph-mon[58107]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:56 smithi149 ceph-mon[58107]: from='client.44380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:56 smithi149 ceph-mon[58107]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:56.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:56 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2514794684' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:45:57.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[64662]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[64662]: from='client.44380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[64662]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2514794684' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:45:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[67080]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[67080]: from='client.44380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:45:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[67080]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:45:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:56 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2514794684' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:45:57.910 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:57 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:58.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:57 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:58.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:57 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:58 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:58 smithi149 ceph-mon[58107]: Upgrade: osd.5 is safe to restart 2024-03-24T11:45:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:58 smithi149 ceph-mon[58107]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:59.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:58 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:59.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:58 smithi145 ceph-mon[67080]: Upgrade: osd.5 is safe to restart 2024-03-24T11:45:59.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:58 smithi145 ceph-mon[67080]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:45:59.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:58 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-24T11:45:59.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:58 smithi145 ceph-mon[64662]: Upgrade: osd.5 is safe to restart 2024-03-24T11:45:59.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:58 smithi145 ceph-mon[64662]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:00.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:59 smithi149 ceph-mon[58107]: Upgrade: Updating osd.5 2024-03-24T11:46:00.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:59 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:00.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:59 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:46:00.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:59 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:00.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:59 smithi149 ceph-mon[58107]: Deploying daemon osd.5 on smithi149 2024-03-24T11:46:00.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:45:59 smithi149 ceph-mon[58107]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[64662]: Upgrade: Updating osd.5 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[64662]: Deploying daemon osd.5 on smithi149 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[64662]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[67080]: Upgrade: Updating osd.5 2024-03-24T11:46:00.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:00.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:46:00.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:00.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[67080]: Deploying daemon osd.5 on smithi149 2024-03-24T11:46:00.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:45:59 smithi145 ceph-mon[67080]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:02.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:02 smithi149 systemd[1]: Stopping Ceph osd.5 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:46:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:02 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:02 smithi149 ceph-mon[58107]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:46:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:02 smithi149 ceph-mon[58107]: osd.5 marked itself down and dead 2024-03-24T11:46:02.975 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[32010]: 2024-03-24T11:46:02.494+0000 7fc64351d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:46:02.976 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[32010]: 2024-03-24T11:46:02.494+0000 7fc64351d700 -1 osd.5 131 *** Got signal Terminated *** 2024-03-24T11:46:02.976 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[32010]: 2024-03-24T11:46:02.494+0000 7fc64351d700 -1 osd.5 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:46:03.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:02 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:02 smithi145 ceph-mon[64662]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:46:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:02 smithi145 ceph-mon[64662]: osd.5 marked itself down and dead 2024-03-24T11:46:03.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:02 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:03.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:02 smithi145 ceph-mon[67080]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:46:03.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:02 smithi145 ceph-mon[67080]: osd.5 marked itself down and dead 2024-03-24T11:46:03.462 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:03 smithi149 podman[63270]: 2024-03-24 11:46:03.1594167 +0000 UTC m=+0.790384802 container died 8d60e7746b35894b9a59bce175afcc7e8c199f990fb90a6d6ed59c36662f0e0c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, version=8, maintainer=Guillaume Abrioux , release=754, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, RELEASE=HEAD) 2024-03-24T11:46:03.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:03 smithi149 podman[63270]: 2024-03-24 11:46:03.552710396 +0000 UTC m=+1.183678491 container cleanup 8d60e7746b35894b9a59bce175afcc7e8c199f990fb90a6d6ed59c36662f0e0c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, version=8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-type=git) 2024-03-24T11:46:03.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:03 smithi149 bash[63270]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5 2024-03-24T11:46:03.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: osdmap e132: 8 total, 7 up, 8 in 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:03.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:03 smithi149 ceph-mon[58107]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:46:04.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: osdmap e132: 8 total, 7 up, 8 in 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[64662]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:46:04.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: osdmap e132: 8 total, 7 up, 8 in 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:03 smithi145 ceph-mon[67080]: failed to encode map e132 with expected crc 2024-03-24T11:46:04.155 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:03 smithi149 podman[63288]: 2024-03-24 11:46:03.838021687 +0000 UTC m=+0.674681285 container remove 8d60e7746b35894b9a59bce175afcc7e8c199f990fb90a6d6ed59c36662f0e0c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, release=754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, ceph=True, RELEASE=HEAD, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8) 2024-03-24T11:46:04.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:04 smithi149 podman[63380]: 2024-03-24 11:46:04.155628122 +0000 UTC m=+0.115422867 container create 2b2fa993ed666eb72e15bd74aeb6d009750db1f796bd2a85dcc8dbbdb8f8f0e0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T11:46:04.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:04 smithi149 podman[63380]: 2024-03-24 11:46:04.05871495 +0000 UTC m=+0.018509714 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:04.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:04 smithi149 podman[63380]: 2024-03-24 11:46:04.391518009 +0000 UTC m=+0.351312766 container init 2b2fa993ed666eb72e15bd74aeb6d009750db1f796bd2a85dcc8dbbdb8f8f0e0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T11:46:04.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:04 smithi149 podman[63380]: 2024-03-24 11:46:04.394592261 +0000 UTC m=+0.354387013 container start 2b2fa993ed666eb72e15bd74aeb6d009750db1f796bd2a85dcc8dbbdb8f8f0e0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:46:04.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:04 smithi149 podman[63380]: 2024-03-24 11:46:04.474708829 +0000 UTC m=+0.434503576 container attach 2b2fa993ed666eb72e15bd74aeb6d009750db1f796bd2a85dcc8dbbdb8f8f0e0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T11:46:04.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:04 smithi149 podman[63380]: 2024-03-24 11:46:04.685389442 +0000 UTC m=+0.645184197 container died 2b2fa993ed666eb72e15bd74aeb6d009750db1f796bd2a85dcc8dbbdb8f8f0e0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-24T11:46:04.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:04 smithi149 ceph-mon[58107]: pgmap v156: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 196 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-24T11:46:04.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:04 smithi149 ceph-mon[58107]: failed to encode map e133 with expected crc 2024-03-24T11:46:04.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:04 smithi149 ceph-mon[58107]: failed to encode map e133 with expected crc 2024-03-24T11:46:04.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:04 smithi149 ceph-mon[58107]: osdmap e133: 8 total, 7 up, 8 in 2024-03-24T11:46:04.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:04 smithi149 ceph-mon[58107]: failed to encode map e133 with expected crc 2024-03-24T11:46:04.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:04.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[67080]: pgmap v156: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 196 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[67080]: failed to encode map e133 with expected crc 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[67080]: failed to encode map e133 with expected crc 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[67080]: osdmap e133: 8 total, 7 up, 8 in 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[67080]: failed to encode map e133 with expected crc 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:05.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[64662]: pgmap v156: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 196 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-24T11:46:05.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[64662]: failed to encode map e133 with expected crc 2024-03-24T11:46:05.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[64662]: failed to encode map e133 with expected crc 2024-03-24T11:46:05.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[64662]: osdmap e133: 8 total, 7 up, 8 in 2024-03-24T11:46:05.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[64662]: failed to encode map e133 with expected crc 2024-03-24T11:46:05.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:05.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:05.459 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 podman[63406]: 2024-03-24 11:46:05.25369399 +0000 UTC m=+0.562857750 container remove 2b2fa993ed666eb72e15bd74aeb6d009750db1f796bd2a85dcc8dbbdb8f8f0e0 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:46:05.460 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-24T11:46:05.460 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 systemd[1]: Stopped Ceph osd.5 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:46:05.460 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.5.service: Consumed 8.441s CPU time. 2024-03-24T11:46:05.460 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 systemd[1]: Starting Ceph osd.5 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:46:05.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 podman[63511]: 2024-03-24 11:46:05.582568815 +0000 UTC m=+0.019727263 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:05.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 podman[63511]: 2024-03-24 11:46:05.696558646 +0000 UTC m=+0.133717093 container create a450406752239cf9c41d78af636f09de6712bfbfc5f90eb0717854d82c5d1c36 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T11:46:05.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:46:05] "GET /metrics HTTP/1.1" 200 38462 "" "Prometheus/2.33.4" 2024-03-24T11:46:05.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:05 smithi145 ceph-mon[67080]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 10 active+undersized, 129 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 103 op/s; 17/627 objects degraded (2.711%) 2024-03-24T11:46:05.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:05 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:05 smithi145 ceph-mon[64662]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 10 active+undersized, 129 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 103 op/s; 17/627 objects degraded (2.711%) 2024-03-24T11:46:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:05 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:06.032 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:05 smithi149 ceph-mon[58107]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 10 active+undersized, 129 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 103 op/s; 17/627 objects degraded (2.711%) 2024-03-24T11:46:06.032 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:05 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:06.033 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 podman[63511]: 2024-03-24 11:46:05.949347691 +0000 UTC m=+0.386506152 container init a450406752239cf9c41d78af636f09de6712bfbfc5f90eb0717854d82c5d1c36 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T11:46:06.033 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:05 smithi149 podman[63511]: 2024-03-24 11:46:05.952863139 +0000 UTC m=+0.390021590 container start a450406752239cf9c41d78af636f09de6712bfbfc5f90eb0717854d82c5d1c36 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:46:06.033 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 podman[63511]: 2024-03-24 11:46:06.032401804 +0000 UTC m=+0.469560270 container attach a450406752239cf9c41d78af636f09de6712bfbfc5f90eb0717854d82c5d1c36 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T11:46:06.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate[63525]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate[63525]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 bash[63511]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 bash[63511]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 bash[63511]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 bash[63511]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 bash[63511]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 bash[63511]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 bash[63511]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate[63525]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate[63525]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate[63525]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate[63525]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T11:46:06.726 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate[63525]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T11:46:06.727 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:06 smithi149 podman[63511]: 2024-03-24 11:46:06.420682077 +0000 UTC m=+0.857840526 container died a450406752239cf9c41d78af636f09de6712bfbfc5f90eb0717854d82c5d1c36 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:46:07.410 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 podman[63658]: 2024-03-24 11:46:07.06139142 +0000 UTC m=+0.634829892 container remove a450406752239cf9c41d78af636f09de6712bfbfc5f90eb0717854d82c5d1c36 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T11:46:07.410 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 podman[63720]: 2024-03-24 11:46:07.185086213 +0000 UTC m=+0.018644378 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:07.410 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 podman[63720]: 2024-03-24 11:46:07.304196177 +0000 UTC m=+0.137754340 container create ef3114caac79b70d5c0b85b32f24aa5d125bff0a1fa8cce0c53d6812cc91f270 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T11:46:07.975 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 podman[63720]: 2024-03-24 11:46:07.623452633 +0000 UTC m=+0.457010821 container init ef3114caac79b70d5c0b85b32f24aa5d125bff0a1fa8cce0c53d6812cc91f270 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T11:46:07.975 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 podman[63720]: 2024-03-24 11:46:07.62653551 +0000 UTC m=+0.460093674 container start ef3114caac79b70d5c0b85b32f24aa5d125bff0a1fa8cce0c53d6812cc91f270 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-24T11:46:07.975 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 ceph-osd[63738]: -- 172.21.15.149:0/2238700659 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562c84ef51e0 con 0x562c84ec6c00 2024-03-24T11:46:07.975 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 bash[63720]: ef3114caac79b70d5c0b85b32f24aa5d125bff0a1fa8cce0c53d6812cc91f270 2024-03-24T11:46:07.976 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:07 smithi149 systemd[1]: Started Ceph osd.5 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:46:08.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:46:08.185+0000 7fb2f1d747c0 -1 Falling back to public interface 2024-03-24T11:46:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:08 smithi149 ceph-mon[58107]: pgmap v159: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 177 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:08 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:08 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:09.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:08 smithi145 ceph-mon[64662]: pgmap v159: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 177 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:08 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:08 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:08 smithi145 ceph-mon[67080]: pgmap v159: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 177 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:08 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:08 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:10.909 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:10 smithi149 ceph-mon[58107]: pgmap v160: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 173 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:11.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:10 smithi145 ceph-mon[67080]: pgmap v160: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 173 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:11.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:10 smithi145 ceph-mon[64662]: pgmap v160: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 173 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:11.225 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:46:10.908+0000 7fb2f1d747c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-24T11:46:11.225 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:46:10.932+0000 7fb2f1d747c0 -1 osd.5 131 log_to_monitors true 2024-03-24T11:46:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:11 smithi149 ceph-mon[58107]: from='osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:46:11.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:11 smithi149 ceph-mon[58107]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:46:11.976 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:46:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:46:11.712+0000 7fb2e31cc700 -1 osd.5 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:46:12.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:11 smithi145 ceph-mon[67080]: from='osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:46:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:11 smithi145 ceph-mon[67080]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:46:12.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:11 smithi145 ceph-mon[64662]: from='osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:46:12.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:11 smithi145 ceph-mon[64662]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:46:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: pgmap v161: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 146 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:46:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: failed to encode map e134 with expected crc 2024-03-24T11:46:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: osdmap e134: 8 total, 7 up, 8 in 2024-03-24T11:46:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: from='osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:12.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:12.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: failed to encode map e134 with expected crc 2024-03-24T11:46:12.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:12.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:12 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:13.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:13.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: pgmap v161: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 146 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: failed to encode map e134 with expected crc 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: osdmap e134: 8 total, 7 up, 8 in 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: from='osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: failed to encode map e134 with expected crc 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: pgmap v161: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 146 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:13.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:46:13.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: failed to encode map e134 with expected crc 2024-03-24T11:46:13.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: osdmap e134: 8 total, 7 up, 8 in 2024-03-24T11:46:13.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: from='osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:13.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:13.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: failed to encode map e134 with expected crc 2024-03-24T11:46:13.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:13.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:12 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:46:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163] boot 2024-03-24T11:46:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: failed to encode map e135 with expected crc 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:13.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: pgmap v164: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 0 B/s wr, 243 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:13.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:13 smithi149 ceph-mon[58107]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163] boot 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: pgmap v164: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 0 B/s wr, 243 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[64662]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: osd.5 [v2:172.21.15.149:6808/2546647163,v1:172.21.15.149:6809/2546647163] boot 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: failed to encode map e135 with expected crc 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: pgmap v164: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 0 B/s wr, 243 op/s; 71/627 objects degraded (11.324%) 2024-03-24T11:46:14.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:13 smithi145 ceph-mon[67080]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:46:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: failed to encode map e136 with expected crc 2024-03-24T11:46:14.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:14 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: failed to encode map e136 with expected crc 2024-03-24T11:46:15.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:14 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:15.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:15 smithi149 ceph-mon[58107]: pgmap v166: 161 pgs: 15 active+undersized+degraded, 34 active+undersized, 112 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 0 B/s wr, 239 op/s; 59/627 objects degraded (9.410%); 0 B/s, 1 objects/s recovering 2024-03-24T11:46:15.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:46:15] "GET /metrics HTTP/1.1" 200 38462 "" "Prometheus/2.33.4" 2024-03-24T11:46:15.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:15 smithi145 ceph-mon[67080]: pgmap v166: 161 pgs: 15 active+undersized+degraded, 34 active+undersized, 112 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 0 B/s wr, 239 op/s; 59/627 objects degraded (9.410%); 0 B/s, 1 objects/s recovering 2024-03-24T11:46:15.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:15 smithi145 ceph-mon[64662]: pgmap v166: 161 pgs: 15 active+undersized+degraded, 34 active+undersized, 112 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 0 B/s wr, 239 op/s; 59/627 objects degraded (9.410%); 0 B/s, 1 objects/s recovering 2024-03-24T11:46:17.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:17 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:46:17.294+0000 7f4ffb2c5700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-24T11:46:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:18.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-24T11:46:18.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:18 smithi149 ceph-mon[58107]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 0 B/s wr, 184 op/s; 0 B/s, 8 objects/s recovering 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:18.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[64662]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 0 B/s wr, 184 op/s; 0 B/s, 8 objects/s recovering 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:18.613 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-24T11:46:18.614 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:18 smithi145 ceph-mon[67080]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 0 B/s wr, 184 op/s; 0 B/s, 8 objects/s recovering 2024-03-24T11:46:19.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:19 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 15 pgs degraded) 2024-03-24T11:46:19.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:19 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:46:19.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:19 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 15 pgs degraded) 2024-03-24T11:46:19.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:19 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:46:19.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:19 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 15 pgs degraded) 2024-03-24T11:46:19.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:19 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:46:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:20 smithi149 ceph-mon[58107]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s; 0 B/s, 6 objects/s recovering 2024-03-24T11:46:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:21.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:20 smithi145 ceph-mon[67080]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s; 0 B/s, 6 objects/s recovering 2024-03-24T11:46:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:20 smithi145 ceph-mon[64662]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s; 0 B/s, 6 objects/s recovering 2024-03-24T11:46:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:22 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:22 smithi149 ceph-mon[58107]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 0 B/s wr, 8 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:46:23.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:22 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:22 smithi145 ceph-mon[67080]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 0 B/s wr, 8 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:46:23.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:22 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:23.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:22 smithi145 ceph-mon[64662]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 0 B/s wr, 8 op/s; 0 B/s, 5 objects/s recovering 2024-03-24T11:46:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:24 smithi149 ceph-mon[58107]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:46:25.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:24 smithi145 ceph-mon[67080]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:46:25.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:24 smithi145 ceph-mon[64662]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:46:25.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:46:25] "GET /metrics HTTP/1.1" 200 38471 "" "Prometheus/2.33.4" 2024-03-24T11:46:25.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:25 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:25.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:25 smithi145 ceph-mon[67080]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:46:25.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:25 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:25.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:25 smithi145 ceph-mon[64662]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:46:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:25 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:25 smithi149 ceph-mon[58107]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T11:46:26.308 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (6m) 84s ago 11m 25.6M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (5m) 14s ago 10m 53.4M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (5m) 84s ago 10m 55.5M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (5m) 14s ago 15m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (5m) 84s ago 17m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (4m) 84s ago 17m 49.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (4m) 14s ago 15m 39.5M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:46:26.953 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (4m) 84s ago 16m 40.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (6m) 84s ago 11m 17.5M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (5m) 14s ago 11m 19.2M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (3m) 84s ago 15m 51.2M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (2m) 84s ago 15m 51.7M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (2m) 84s ago 14m 41.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (89s) 84s ago 14m 12.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (54s) 14s ago 13m 55.4M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 8a8a001df0e1 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (19s) 14s ago 13m 13.8M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 ef3114caac79 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (13m) 14s ago 13m 57.9M 1454M 17.2.0 e1d6a67b021e 95f1220be232 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (12m) 14s ago 12m 62.3M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (5m) 14s ago 11m 68.9M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (10m) 84s ago 10m 72.2M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (10m) 14s ago 10m 76.2M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (10m) 84s ago 10m 72.8M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:46:26.954 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (10m) 14s ago 10m 75.0M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:46:26.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:26 smithi145 ceph-mon[64662]: from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:26.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:26 smithi145 ceph-mon[64662]: from='client.44401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:26.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:26 smithi145 ceph-mon[67080]: from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:26.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:26 smithi145 ceph-mon[67080]: from='client.44401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:27.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:26 smithi149 ceph-mon[58107]: from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:27.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:26 smithi149 ceph-mon[58107]: from='client.44401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:46:27.348 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 6 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 11 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:46:27.349 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "11/23 daemons upgraded", 2024-03-24T11:46:27.678 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:46:27.679 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:46:27.679 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:46:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:27 smithi149 ceph-mon[58107]: from='client.44407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:27 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/1206617468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:27 smithi149 ceph-mon[58107]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 3 objects/s recovering 2024-03-24T11:46:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:27 smithi149 ceph-mon[58107]: from='client.44419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:28.070 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[64662]: from='client.44407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1206617468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[64662]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 3 objects/s recovering 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[64662]: from='client.44419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[67080]: from='client.44407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/1206617468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[67080]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 3 objects/s recovering 2024-03-24T11:46:28.071 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:27 smithi145 ceph-mon[67080]: from='client.44419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:29.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:28 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1556877663' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:46:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:28 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/1556877663' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:46:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:28 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/1556877663' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:46:30.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:29 smithi145 ceph-mon[64662]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:29 smithi145 ceph-mon[67080]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:30.126 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:29 smithi149 ceph-mon[58107]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:32.933 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:32 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:32.934 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:32 smithi149 ceph-mon[58107]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:32.934 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:32 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:33.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:32 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:33.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:32 smithi145 ceph-mon[67080]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:33.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:32 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:33.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:32 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:33.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:32 smithi145 ceph-mon[64662]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:33.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:32 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:33.937 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:33 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:33.937 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:33 smithi149 ceph-mon[58107]: Upgrade: osd.6 is safe to restart 2024-03-24T11:46:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:33 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:33 smithi145 ceph-mon[64662]: Upgrade: osd.6 is safe to restart 2024-03-24T11:46:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:33 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-24T11:46:34.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:33 smithi145 ceph-mon[67080]: Upgrade: osd.6 is safe to restart 2024-03-24T11:46:34.919 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:34 smithi149 ceph-mon[58107]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:34.919 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:34.919 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:46:34.919 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:34.919 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[64662]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:46:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[67080]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:46:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:35.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:46:35] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.33.4" 2024-03-24T11:46:35.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:35 smithi145 ceph-mon[67080]: Upgrade: Updating osd.6 2024-03-24T11:46:35.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:35 smithi145 ceph-mon[67080]: Deploying daemon osd.6 on smithi149 2024-03-24T11:46:35.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:35 smithi145 ceph-mon[64662]: Upgrade: Updating osd.6 2024-03-24T11:46:35.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:35 smithi145 ceph-mon[64662]: Deploying daemon osd.6 on smithi149 2024-03-24T11:46:35.964 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:35 smithi149 ceph-mon[58107]: Upgrade: Updating osd.6 2024-03-24T11:46:35.964 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:35 smithi149 ceph-mon[58107]: Deploying daemon osd.6 on smithi149 2024-03-24T11:46:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:36 smithi149 ceph-mon[58107]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:37.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:36 smithi145 ceph-mon[67080]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:37.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:36 smithi145 ceph-mon[64662]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:46:37.700 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:37 smithi149 systemd[1]: Stopping Ceph osd.6 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:46:37.700 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[34891]: 2024-03-24T11:46:37.642+0000 7fa735bc9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:46:37.700 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[34891]: 2024-03-24T11:46:37.642+0000 7fa735bc9700 -1 osd.6 136 *** Got signal Terminated *** 2024-03-24T11:46:37.700 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:37 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[34891]: 2024-03-24T11:46:37.642+0000 7fa735bc9700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:46:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:37 smithi149 ceph-mon[58107]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:37 smithi149 ceph-mon[58107]: osd.6 marked itself down and dead 2024-03-24T11:46:38.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:37 smithi145 ceph-mon[67080]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:38.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:37 smithi145 ceph-mon[67080]: osd.6 marked itself down and dead 2024-03-24T11:46:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:37 smithi145 ceph-mon[64662]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:46:38.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:37 smithi145 ceph-mon[64662]: osd.6 marked itself down and dead 2024-03-24T11:46:38.377 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:38 smithi149 podman[65598]: 2024-03-24 11:46:38.085014401 +0000 UTC m=+0.540331630 container died 95f1220be232c3a088d20c13f7424ee481fa3a7f954510fe2721b30c6dad0b25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, ceph=True, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T11:46:38.694 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:38 smithi149 podman[65598]: 2024-03-24 11:46:38.484511334 +0000 UTC m=+0.939828560 container cleanup 95f1220be232c3a088d20c13f7424ee481fa3a7f954510fe2721b30c6dad0b25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, io.openshift.tags=base centos centos-stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.expose-services=, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc.) 2024-03-24T11:46:38.694 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:38 smithi149 bash[65598]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6 2024-03-24T11:46:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:38 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:46:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:38 smithi149 ceph-mon[58107]: failed to encode map e137 with expected crc 2024-03-24T11:46:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:38 smithi149 ceph-mon[58107]: osdmap e137: 8 total, 7 up, 8 in 2024-03-24T11:46:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:38 smithi149 ceph-mon[58107]: failed to encode map e137 with expected crc 2024-03-24T11:46:38.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:38 smithi149 podman[65615]: 2024-03-24 11:46:38.694100944 +0000 UTC m=+0.612398465 container remove 95f1220be232c3a088d20c13f7424ee481fa3a7f954510fe2721b30c6dad0b25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, release=754, name=centos-stream, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, ceph=True) 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[67080]: failed to encode map e137 with expected crc 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[67080]: osdmap e137: 8 total, 7 up, 8 in 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[67080]: failed to encode map e137 with expected crc 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[64662]: failed to encode map e137 with expected crc 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[64662]: osdmap e137: 8 total, 7 up, 8 in 2024-03-24T11:46:39.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:38 smithi145 ceph-mon[64662]: failed to encode map e137 with expected crc 2024-03-24T11:46:39.447 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:39 smithi149 podman[65711]: 2024-03-24 11:46:39.014421966 +0000 UTC m=+0.019104710 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:39.447 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:39 smithi149 podman[65711]: 2024-03-24 11:46:39.136735958 +0000 UTC m=+0.141418702 container create 6ff0c216d0111481784477ba654d1ae6cf7a900ed4f141aea4dbe808c244d342 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:46:39.448 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:39 smithi149 podman[65711]: 2024-03-24 11:46:39.34771295 +0000 UTC m=+0.352395708 container init 6ff0c216d0111481784477ba654d1ae6cf7a900ed4f141aea4dbe808c244d342 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T11:46:39.448 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:39 smithi149 podman[65711]: 2024-03-24 11:46:39.350791429 +0000 UTC m=+0.355474173 container start 6ff0c216d0111481784477ba654d1ae6cf7a900ed4f141aea4dbe808c244d342 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:46:39.448 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:39 smithi149 podman[65711]: 2024-03-24 11:46:39.447602043 +0000 UTC m=+0.452284796 container attach 6ff0c216d0111481784477ba654d1ae6cf7a900ed4f141aea4dbe808c244d342 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:46:39.718 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:39 smithi149 podman[65711]: 2024-03-24 11:46:39.645371727 +0000 UTC m=+0.650054473 container died 6ff0c216d0111481784477ba654d1ae6cf7a900ed4f141aea4dbe808c244d342 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T11:46:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: osdmap e138: 8 total, 7 up, 8 in 2024-03-24T11:46:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: failed to encode map e138 with expected crc 2024-03-24T11:46:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: failed to encode map e138 with expected crc 2024-03-24T11:46:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: failed to encode map e138 with expected crc 2024-03-24T11:46:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: failed to encode map e138 with expected crc 2024-03-24T11:46:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: failed to encode map e138 with expected crc 2024-03-24T11:46:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: failed to encode map e138 with expected crc 2024-03-24T11:46:39.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:39 smithi149 ceph-mon[58107]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:46:40.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: osdmap e138: 8 total, 7 up, 8 in 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[64662]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: osdmap e138: 8 total, 7 up, 8 in 2024-03-24T11:46:40.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: failed to encode map e138 with expected crc 2024-03-24T11:46:40.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:39 smithi145 ceph-mon[67080]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:46:40.725 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 podman[65737]: 2024-03-24 11:46:40.285189539 +0000 UTC m=+0.634527472 container remove 6ff0c216d0111481784477ba654d1ae6cf7a900ed4f141aea4dbe808c244d342 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:46:40.725 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-24T11:46:40.725 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 systemd[1]: Stopped Ceph osd.6 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:46:40.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.6.service: Consumed 7.967s CPU time. 2024-03-24T11:46:40.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 systemd[1]: Starting Ceph osd.6 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:46:41.038 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 podman[65844]: 2024-03-24 11:46:40.632203763 +0000 UTC m=+0.018577748 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:41.038 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 podman[65844]: 2024-03-24 11:46:40.736208783 +0000 UTC m=+0.122582795 container create 947bdaaa561e77531ab362652cd60df1158a4b9c9357fcc17831c9b31272ded5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T11:46:41.039 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 podman[65844]: 2024-03-24 11:46:40.955273855 +0000 UTC m=+0.341647826 container init 947bdaaa561e77531ab362652cd60df1158a4b9c9357fcc17831c9b31272ded5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T11:46:41.039 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:40 smithi149 podman[65844]: 2024-03-24 11:46:40.958804501 +0000 UTC m=+0.345178476 container start 947bdaaa561e77531ab362652cd60df1158a4b9c9357fcc17831c9b31272ded5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T11:46:41.039 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 podman[65844]: 2024-03-24 11:46:41.038542468 +0000 UTC m=+0.424916457 container attach 947bdaaa561e77531ab362652cd60df1158a4b9c9357fcc17831c9b31272ded5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T11:46:41.725 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate[65858]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate[65858]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 bash[65844]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 bash[65844]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 bash[65844]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 bash[65844]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 bash[65844]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 bash[65844]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 bash[65844]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate[65858]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate[65858]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate[65858]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-24T11:46:41.726 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate[65858]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T11:46:41.727 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate[65858]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-24T11:46:41.727 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:41 smithi149 podman[65844]: 2024-03-24 11:46:41.424407159 +0000 UTC m=+0.810781126 container died 947bdaaa561e77531ab362652cd60df1158a4b9c9357fcc17831c9b31272ded5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:46:42.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:42.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-mon[58107]: pgmap v181: 161 pgs: 3 active+undersized+degraded, 12 stale+active+clean, 7 active+undersized, 139 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 15/627 objects degraded (2.392%) 2024-03-24T11:46:42.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:41 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 15/627 objects degraded (2.392%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:42.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:41 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:42.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:41 smithi145 ceph-mon[67080]: pgmap v181: 161 pgs: 3 active+undersized+degraded, 12 stale+active+clean, 7 active+undersized, 139 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 15/627 objects degraded (2.392%) 2024-03-24T11:46:42.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:41 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 15/627 objects degraded (2.392%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:42.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:41 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:42.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:41 smithi145 ceph-mon[64662]: pgmap v181: 161 pgs: 3 active+undersized+degraded, 12 stale+active+clean, 7 active+undersized, 139 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 15/627 objects degraded (2.392%) 2024-03-24T11:46:42.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:41 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 15/627 objects degraded (2.392%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:42.437 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 podman[65975]: 2024-03-24 11:46:42.051094722 +0000 UTC m=+0.621865177 container remove 947bdaaa561e77531ab362652cd60df1158a4b9c9357fcc17831c9b31272ded5 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T11:46:42.437 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 podman[66040]: 2024-03-24 11:46:42.166326795 +0000 UTC m=+0.017665590 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:42.437 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 podman[66040]: 2024-03-24 11:46:42.285536624 +0000 UTC m=+0.136875460 container create 1223bf8d794a7da94f0553713f727797e16350c9c4372807ef3e8fd1515efc78 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T11:46:42.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 podman[66040]: 2024-03-24 11:46:42.563648436 +0000 UTC m=+0.414987245 container init 1223bf8d794a7da94f0553713f727797e16350c9c4372807ef3e8fd1515efc78 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b) 2024-03-24T11:46:42.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 podman[66040]: 2024-03-24 11:46:42.566925306 +0000 UTC m=+0.418264103 container start 1223bf8d794a7da94f0553713f727797e16350c9c4372807ef3e8fd1515efc78 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:46:42.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 ceph-osd[66059]: -- 172.21.15.149:0/130446289 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558a1b5871e0 con 0x558a1b558c00 2024-03-24T11:46:42.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 bash[66040]: 1223bf8d794a7da94f0553713f727797e16350c9c4372807ef3e8fd1515efc78 2024-03-24T11:46:42.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:42 smithi149 systemd[1]: Started Ceph osd.6 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:46:43.475 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:43 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:46:43.107+0000 7efd4819b7c0 -1 Falling back to public interface 2024-03-24T11:46:43.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:43 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:43.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:43 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:43.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:43 smithi149 ceph-mon[58107]: pgmap v182: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:44.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:43 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:44.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:43 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:44.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:43 smithi145 ceph-mon[64662]: pgmap v182: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:44.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:43 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:44.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:43 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:44.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:43 smithi145 ceph-mon[67080]: pgmap v182: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:45.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:46:45] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.33.4" 2024-03-24T11:46:46.026 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:46:45.843+0000 7efd4819b7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-24T11:46:46.026 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:45 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:46:45.879+0000 7efd4819b7c0 -1 osd.6 136 log_to_monitors true 2024-03-24T11:46:46.698 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:46 smithi149 ceph-mon[58107]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:46.698 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:46 smithi149 ceph-mon[58107]: from='osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:46:46.699 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:46 smithi149 ceph-mon[58107]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:46:46.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:46:46 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:46:46.725+0000 7efd395f3700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:46:47.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:46 smithi145 ceph-mon[67080]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:47.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:46 smithi145 ceph-mon[67080]: from='osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:46:47.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:46 smithi145 ceph-mon[67080]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:46:47.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:46 smithi145 ceph-mon[64662]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:47.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:46 smithi145 ceph-mon[64662]: from='osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:46:47.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:46 smithi145 ceph-mon[64662]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:46:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:46:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: failed to encode map e139 with expected crc 2024-03-24T11:46:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: osdmap e139: 8 total, 7 up, 8 in 2024-03-24T11:46:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: from='osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:47.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:47.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:47 smithi149 ceph-mon[58107]: pgmap v185: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:48.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:46:48.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: failed to encode map e139 with expected crc 2024-03-24T11:46:48.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: osdmap e139: 8 total, 7 up, 8 in 2024-03-24T11:46:48.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: from='osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:48.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[67080]: pgmap v185: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: failed to encode map e139 with expected crc 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: osdmap e139: 8 total, 7 up, 8 in 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: from='osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:48.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:47 smithi145 ceph-mon[64662]: pgmap v185: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-24T11:46:48.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:46:48.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: failed to encode map e140 with expected crc 2024-03-24T11:46:48.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779] boot 2024-03-24T11:46:48.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:46:48.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: failed to encode map e140 with expected crc 2024-03-24T11:46:48.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:46:48.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: failed to encode map e140 with expected crc 2024-03-24T11:46:48.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: failed to encode map e140 with expected crc 2024-03-24T11:46:48.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:48.977 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:48 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779] boot 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:49.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: osd.6 [v2:172.21.15.149:6816/3640756779,v1:172.21.15.149:6817/3640756779] boot 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: failed to encode map e140 with expected crc 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:49.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:48 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:49 smithi149 ceph-mon[58107]: failed to encode map e141 with expected crc 2024-03-24T11:46:49.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:49 smithi149 ceph-mon[58107]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:46:49.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:49 smithi149 ceph-mon[58107]: failed to encode map e141 with expected crc 2024-03-24T11:46:49.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:49 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:49.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:49 smithi149 ceph-mon[58107]: pgmap v188: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/627 objects degraded (12.121%) 2024-03-24T11:46:49.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:49.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[67080]: failed to encode map e141 with expected crc 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[67080]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[67080]: failed to encode map e141 with expected crc 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[67080]: pgmap v188: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/627 objects degraded (12.121%) 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[64662]: failed to encode map e141 with expected crc 2024-03-24T11:46:50.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[64662]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:46:50.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[64662]: failed to encode map e141 with expected crc 2024-03-24T11:46:50.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 18 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:50.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[64662]: pgmap v188: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/627 objects degraded (12.121%) 2024-03-24T11:46:50.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:50.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:46:52.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:51 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:46:51.750+0000 7f4ffb2c5700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: pgmap v189: 161 pgs: 15 active+undersized+degraded, 29 active+undersized, 117 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:52.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: pgmap v189: 161 pgs: 15 active+undersized+degraded, 29 active+undersized, 117 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:52.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.864 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:46:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:46:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: pgmap v189: 161 pgs: 15 active+undersized+degraded, 29 active+undersized, 117 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:52.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:46:53.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:53 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:46:53.892 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:53 smithi145 ceph-mon[67080]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:46:53.892 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:53 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:46:53.892 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:53 smithi145 ceph-mon[64662]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:46:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:53 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:46:53.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:53 smithi149 ceph-mon[58107]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-24T11:46:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:54 smithi149 ceph-mon[58107]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:46:54.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:54 smithi149 ceph-mon[58107]: Health check update: Degraded data redundancy: 61/627 objects degraded (9.729%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:55.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:54 smithi145 ceph-mon[67080]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:46:55.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:54 smithi145 ceph-mon[67080]: Health check update: Degraded data redundancy: 61/627 objects degraded (9.729%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:55.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:54 smithi145 ceph-mon[64662]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:46:55.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:54 smithi145 ceph-mon[64662]: Health check update: Degraded data redundancy: 61/627 objects degraded (9.729%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T11:46:55.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:55 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/627 objects degraded (9.729%), 15 pgs degraded) 2024-03-24T11:46:55.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:55 smithi145 ceph-mon[64662]: Cluster is now healthy 2024-03-24T11:46:55.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:46:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:46:55] "GET /metrics HTTP/1.1" 200 38342 "" "Prometheus/2.33.4" 2024-03-24T11:46:55.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:55 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/627 objects degraded (9.729%), 15 pgs degraded) 2024-03-24T11:46:55.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:55 smithi145 ceph-mon[67080]: Cluster is now healthy 2024-03-24T11:46:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:55 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/627 objects degraded (9.729%), 15 pgs degraded) 2024-03-24T11:46:55.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:55 smithi149 ceph-mon[58107]: Cluster is now healthy 2024-03-24T11:46:56.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:56 smithi149 ceph-mon[58107]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:46:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:56 smithi145 ceph-mon[64662]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:46:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:56 smithi145 ceph-mon[67080]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:46:58.424 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:46:58.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:58 smithi149 ceph-mon[58107]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-03-24T11:46:59.068 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:58 smithi145 ceph-mon[67080]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-03-24T11:46:59.068 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:58 smithi145 ceph-mon[64662]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (6m) 116s ago 11m 25.6M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (6m) 11s ago 11m 53.4M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (5m) 116s ago 10m 55.5M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (5m) 11s ago 16m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (6m) 116s ago 17m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (5m) 116s ago 18m 49.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (4m) 11s ago 16m 40.9M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (4m) 116s ago 16m 40.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (7m) 116s ago 12m 17.5M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (6m) 11s ago 12m 18.5M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (3m) 116s ago 15m 51.2M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (3m) 116s ago 15m 51.7M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:46:59.069 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (2m) 116s ago 15m 41.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (2m) 116s ago 14m 12.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (86s) 11s ago 14m 57.7M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 8a8a001df0e1 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (51s) 11s ago 14m 45.1M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 ef3114caac79 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (16s) 11s ago 13m 12.7M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 1223bf8d794a 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (13m) 11s ago 13m 64.2M 1454M 17.2.0 e1d6a67b021e 767f2fa6c3e1 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (5m) 11s ago 12m 69.0M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (10m) 116s ago 10m 72.2M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (10m) 11s ago 10m 76.4M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (10m) 116s ago 10m 72.8M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:46:59.070 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (10m) 11s ago 10m 75.5M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:46:59.459 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:46:59.459 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:46:59.459 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:46:59.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:59.459 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:46:59.459 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:46:59.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 7 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 12 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:46:59.460 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:46:59.794 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:46:59.794 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:46:59.794 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:46:59.794 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:46:59.794 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:46:59.795 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:46:59.795 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:46:59.795 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:46:59.795 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "12/23 daemons upgraded", 2024-03-24T11:46:59.795 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading osd daemons", 2024-03-24T11:46:59.795 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:46:59.795 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:46:59.795 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:59 smithi145 ceph-mon[67080]: from='client.44437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:59.795 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:46:59 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2287380831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:59.795 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:59 smithi145 ceph-mon[64662]: from='client.44437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:59.796 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:46:59 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2287380831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:46:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:59 smithi149 ceph-mon[58107]: from='client.44437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:59.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:46:59 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2287380831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:00.180 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:47:00.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:00 smithi149 ceph-mon[58107]: from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:00.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:00 smithi149 ceph-mon[58107]: from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:00.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:00 smithi149 ceph-mon[58107]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-03-24T11:47:00.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:00.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:00 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3311339747' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:01.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[67080]: from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[67080]: from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[67080]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3311339747' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[64662]: from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[64662]: from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[64662]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:01.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:00 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3311339747' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:01 smithi149 ceph-mon[58107]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:02.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:01 smithi145 ceph-mon[67080]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:02.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:01 smithi145 ceph-mon[64662]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:02 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:02 smithi149 ceph-mon[58107]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:47:03.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:02 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:03.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:02 smithi145 ceph-mon[67080]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:47:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:02 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:03.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:02 smithi145 ceph-mon[64662]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-24T11:47:03.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:03 smithi149 ceph-mon[58107]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:03 smithi145 ceph-mon[67080]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:03 smithi145 ceph-mon[64662]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:05.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:05 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:05.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:05 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:05 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:05 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:05.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:47:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:47:05] "GET /metrics HTTP/1.1" 200 38483 "" "Prometheus/2.33.4" 2024-03-24T11:47:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:05 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:05 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:06.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:06 smithi149 ceph-mon[58107]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:07.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:06 smithi145 ceph-mon[67080]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:07.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:06 smithi145 ceph-mon[64662]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:07 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:47:08.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:07 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:47:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:07 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:47:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:08 smithi149 ceph-mon[58107]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:47:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:08 smithi149 ceph-mon[58107]: Upgrade: osd.7 is safe to restart 2024-03-24T11:47:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:08 smithi149 ceph-mon[58107]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:08 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:08 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:47:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:08 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:09.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[67080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[67080]: Upgrade: osd.7 is safe to restart 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[67080]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[64662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[64662]: Upgrade: osd.7 is safe to restart 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[64662]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:47:09.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:08 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:09 smithi149 ceph-mon[58107]: Upgrade: Updating osd.7 2024-03-24T11:47:09.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:09 smithi149 ceph-mon[58107]: Deploying daemon osd.7 on smithi149 2024-03-24T11:47:10.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:09 smithi145 ceph-mon[64662]: Upgrade: Updating osd.7 2024-03-24T11:47:10.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:09 smithi145 ceph-mon[64662]: Deploying daemon osd.7 on smithi149 2024-03-24T11:47:10.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:09 smithi145 ceph-mon[67080]: Upgrade: Updating osd.7 2024-03-24T11:47:10.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:09 smithi145 ceph-mon[67080]: Deploying daemon osd.7 on smithi149 2024-03-24T11:47:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:10 smithi149 ceph-mon[58107]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:11.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:10 smithi145 ceph-mon[64662]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:10 smithi145 ceph-mon[67080]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:11.974 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:11 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:11.974 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:11 smithi149 ceph-mon[58107]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:11.974 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:11 smithi149 systemd[1]: Stopping Ceph osd.7 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:47:12.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:11 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:12.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:11 smithi145 ceph-mon[64662]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:11 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:11 smithi145 ceph-mon[67080]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:47:12.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-24T11:47:11.972+0000 7fd8799d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:47:12.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-24T11:47:11.972+0000 7fd8799d1700 -1 osd.7 141 *** Got signal Terminated *** 2024-03-24T11:47:12.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-24T11:47:11.972+0000 7fd8799d1700 -1 osd.7 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:47:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:12 smithi149 ceph-mon[58107]: osd.7 marked itself down and dead 2024-03-24T11:47:13.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:12 smithi145 ceph-mon[64662]: osd.7 marked itself down and dead 2024-03-24T11:47:13.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:12 smithi145 ceph-mon[67080]: osd.7 marked itself down and dead 2024-03-24T11:47:13.349 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:13 smithi149 podman[67933]: 2024-03-24 11:47:13.074760355 +0000 UTC m=+1.209527830 container died 767f2fa6c3e1eb6247d27d90bdb443ff7ea283ace24cc32275715aa684723ea3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux ) 2024-03-24T11:47:13.706 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:13 smithi149 podman[67933]: 2024-03-24 11:47:13.447404241 +0000 UTC m=+1.582171708 container cleanup 767f2fa6c3e1eb6247d27d90bdb443ff7ea283ace24cc32275715aa684723ea3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, RELEASE=HEAD, com.redhat.component=centos-stream-container) 2024-03-24T11:47:13.707 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:13 smithi149 bash[67933]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7 2024-03-24T11:47:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:13 smithi149 ceph-mon[58107]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:47:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:13 smithi149 ceph-mon[58107]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-24T11:47:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:13 smithi149 ceph-mon[58107]: osdmap e142: 8 total, 7 up, 8 in 2024-03-24T11:47:13.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:13 smithi149 ceph-mon[58107]: pgmap v201: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:47:13.976 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:13 smithi149 podman[67951]: 2024-03-24 11:47:13.70685405 +0000 UTC m=+0.625872957 container remove 767f2fa6c3e1eb6247d27d90bdb443ff7ea283ace24cc32275715aa684723ea3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , version=8, GIT_BRANCH=HEAD, ceph=True, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream) 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[67080]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[67080]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[67080]: osdmap e142: 8 total, 7 up, 8 in 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[67080]: pgmap v201: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[64662]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[64662]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[64662]: osdmap e142: 8 total, 7 up, 8 in 2024-03-24T11:47:14.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:13 smithi145 ceph-mon[64662]: pgmap v201: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:47:14.307 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:14 smithi149 podman[68044]: 2024-03-24 11:47:13.958889743 +0000 UTC m=+0.017971330 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:14.307 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:14 smithi149 podman[68044]: 2024-03-24 11:47:14.091524026 +0000 UTC m=+0.150605616 container create f0304bbf1a3b5808829c1eeadc5848a09a4d2ec244ac96b8658af69e74af7fc2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:47:14.710 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:14 smithi149 podman[68044]: 2024-03-24 11:47:14.419531606 +0000 UTC m=+0.478613208 container init f0304bbf1a3b5808829c1eeadc5848a09a4d2ec244ac96b8658af69e74af7fc2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T11:47:14.710 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:14 smithi149 podman[68044]: 2024-03-24 11:47:14.422768725 +0000 UTC m=+0.481850320 container start f0304bbf1a3b5808829c1eeadc5848a09a4d2ec244ac96b8658af69e74af7fc2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T11:47:14.710 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:14 smithi149 podman[68044]: 2024-03-24 11:47:14.50237535 +0000 UTC m=+0.561456955 container attach f0304bbf1a3b5808829c1eeadc5848a09a4d2ec244ac96b8658af69e74af7fc2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T11:47:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:14 smithi149 ceph-mon[58107]: osdmap e143: 8 total, 7 up, 8 in 2024-03-24T11:47:14.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:14 smithi149 podman[68044]: 2024-03-24 11:47:14.709823205 +0000 UTC m=+0.768904804 container died f0304bbf1a3b5808829c1eeadc5848a09a4d2ec244ac96b8658af69e74af7fc2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:47:15.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:14 smithi145 ceph-mon[64662]: osdmap e143: 8 total, 7 up, 8 in 2024-03-24T11:47:15.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:14 smithi145 ceph-mon[67080]: osdmap e143: 8 total, 7 up, 8 in 2024-03-24T11:47:15.630 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:15 smithi149 podman[68070]: 2024-03-24 11:47:15.406522806 +0000 UTC m=+0.690328749 container remove f0304bbf1a3b5808829c1eeadc5848a09a4d2ec244ac96b8658af69e74af7fc2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T11:47:15.630 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:15 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-24T11:47:15.630 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:15 smithi149 systemd[1]: Stopped Ceph osd.7 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:47:15.630 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:15 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.7.service: Consumed 10.645s CPU time. 2024-03-24T11:47:15.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:15 smithi145 ceph-mon[67080]: pgmap v203: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:47:15.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:15 smithi145 ceph-mon[64662]: pgmap v203: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:47:15.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:47:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:47:15] "GET /metrics HTTP/1.1" 200 38483 "" "Prometheus/2.33.4" 2024-03-24T11:47:15.964 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:15 smithi149 ceph-mon[58107]: pgmap v203: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:47:15.964 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:15 smithi149 systemd[1]: Starting Ceph osd.7 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:47:15.964 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:15 smithi149 podman[68174]: 2024-03-24 11:47:15.753600796 +0000 UTC m=+0.018603283 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:15.965 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:15 smithi149 podman[68174]: 2024-03-24 11:47:15.866459829 +0000 UTC m=+0.131462303 container create 8bafcea031fed07f92ab61c0e906c7445a93e28b788a1090f0ba53524d575916 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T11:47:16.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 podman[68174]: 2024-03-24 11:47:16.085543134 +0000 UTC m=+0.350545620 container init 8bafcea031fed07f92ab61c0e906c7445a93e28b788a1090f0ba53524d575916 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T11:47:16.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 podman[68174]: 2024-03-24 11:47:16.088742056 +0000 UTC m=+0.353744534 container start 8bafcea031fed07f92ab61c0e906c7445a93e28b788a1090f0ba53524d575916 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:47:16.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 podman[68174]: 2024-03-24 11:47:16.168823366 +0000 UTC m=+0.433825861 container attach 8bafcea031fed07f92ab61c0e906c7445a93e28b788a1090f0ba53524d575916 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate[68189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate[68189]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 bash[68174]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 bash[68174]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 bash[68174]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 bash[68174]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 bash[68174]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 bash[68174]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 bash[68174]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-24T11:47:16.848 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate[68189]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-24T11:47:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate[68189]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-24T11:47:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate[68189]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-24T11:47:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate[68189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T11:47:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate[68189]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-24T11:47:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:16 smithi149 podman[68174]: 2024-03-24 11:47:16.552926815 +0000 UTC m=+0.817929287 container died 8bafcea031fed07f92ab61c0e906c7445a93e28b788a1090f0ba53524d575916 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T11:47:17.588 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 podman[68307]: 2024-03-24 11:47:17.256409941 +0000 UTC m=+0.698481903 container remove 8bafcea031fed07f92ab61c0e906c7445a93e28b788a1090f0ba53524d575916 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T11:47:17.588 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 podman[68372]: 2024-03-24 11:47:17.377968746 +0000 UTC m=+0.018490618 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:47:17.588 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 podman[68372]: 2024-03-24 11:47:17.48232886 +0000 UTC m=+0.122850726 container create f5d48891e80cbda39c86f42e17bca4901ff80b24f7e3a892bd31739eafc87296 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T11:47:18.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:17 smithi145 ceph-mon[64662]: pgmap v204: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:18.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:17 smithi145 ceph-mon[64662]: Health check failed: Degraded data redundancy: 80/627 objects degraded (12.759%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:47:18.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:17 smithi145 ceph-mon[67080]: pgmap v204: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:18.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:17 smithi145 ceph-mon[67080]: Health check failed: Degraded data redundancy: 80/627 objects degraded (12.759%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:47:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:17 smithi149 ceph-mon[58107]: pgmap v204: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:18.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:17 smithi149 ceph-mon[58107]: Health check failed: Degraded data redundancy: 80/627 objects degraded (12.759%), 23 pgs degraded (PG_DEGRADED) 2024-03-24T11:47:18.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 podman[68372]: 2024-03-24 11:47:17.75158034 +0000 UTC m=+0.392102208 container init f5d48891e80cbda39c86f42e17bca4901ff80b24f7e3a892bd31739eafc87296 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T11:47:18.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 podman[68372]: 2024-03-24 11:47:17.754782511 +0000 UTC m=+0.395304379 container start f5d48891e80cbda39c86f42e17bca4901ff80b24f7e3a892bd31739eafc87296 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T11:47:18.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 ceph-osd[68390]: -- 172.21.15.149:0/862078074 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558da7c8b1e0 con 0x558da7c5d400 2024-03-24T11:47:18.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 bash[68372]: f5d48891e80cbda39c86f42e17bca4901ff80b24f7e3a892bd31739eafc87296 2024-03-24T11:47:18.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:17 smithi149 systemd[1]: Started Ceph osd.7 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:47:18.698 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:47:18.294+0000 7f02e1c617c0 -1 Falling back to public interface 2024-03-24T11:47:18.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:18.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:18 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:19.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:19.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:18 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:19.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:19.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:18 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:20 smithi149 ceph-mon[58107]: pgmap v205: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:20 smithi145 ceph-mon[64662]: pgmap v205: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:20 smithi145 ceph-mon[67080]: pgmap v205: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:21.241 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:21 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:47:21.030+0000 7f02e1c617c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-24T11:47:21.241 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:21 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:47:21.062+0000 7f02e1c617c0 -1 osd.7 141 log_to_monitors true 2024-03-24T11:47:21.934 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:21 smithi149 ceph-mon[58107]: from='osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:47:21.935 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:21 smithi149 ceph-mon[58107]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:47:21.935 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:47:21 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:47:21.670+0000 7f02d30b9700 -1 osd.7 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:47:22.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:21 smithi145 ceph-mon[64662]: from='osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:47:22.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:21 smithi145 ceph-mon[64662]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:47:22.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:21 smithi145 ceph-mon[67080]: from='osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:47:22.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:21 smithi145 ceph-mon[67080]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:47:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: pgmap v206: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:47:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: osdmap e144: 8 total, 7 up, 8 in 2024-03-24T11:47:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: from='osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:47:22.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:47:22.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:22.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:22 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: pgmap v206: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: osdmap e144: 8 total, 7 up, 8 in 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: from='osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:23.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:23.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: pgmap v206: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:23.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:47:23.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: osdmap e144: 8 total, 7 up, 8 in 2024-03-24T11:47:23.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: from='osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:47:23.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-24T11:47:23.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:23.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:22 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:23.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:23 smithi149 ceph-mon[58107]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:47:23.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:23 smithi149 ceph-mon[58107]: osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322] boot 2024-03-24T11:47:23.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:23 smithi149 ceph-mon[58107]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:47:23.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:23 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:47:23.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:23 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:23.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:23 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:23.929 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:23 smithi149 ceph-mon[58107]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:47:24.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[64662]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[64662]: osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322] boot 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[64662]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[64662]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[67080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[67080]: osd.7 [v2:172.21.15.149:6824/2376058322,v1:172.21.15.149:6825/2376058322] boot 2024-03-24T11:47:24.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[67080]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:47:24.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:47:24.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:24.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:24.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:23 smithi145 ceph-mon[67080]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:47:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:24 smithi149 ceph-mon[58107]: pgmap v209: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:25.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:24 smithi145 ceph-mon[64662]: pgmap v209: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:25.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:24 smithi145 ceph-mon[67080]: pgmap v209: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-24T11:47:25.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:47:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:47:25] "GET /metrics HTTP/1.1" 200 38561 "" "Prometheus/2.33.4" 2024-03-24T11:47:26.809 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:26 smithi149 ceph-mon[58107]: pgmap v211: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-03-24T11:47:27.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:26 smithi145 ceph-mon[64662]: pgmap v211: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-03-24T11:47:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:26 smithi145 ceph-mon[67080]: pgmap v211: 161 pgs: 23 active+undersized+degraded, 46 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: Upgrade: Setting container_image for all osd 2024-03-24T11:47:28.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-24T11:47:28.113 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.114 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: Upgrade: Setting container_image for all osd 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-24T11:47:28.115 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: Upgrade: Setting require_osd_release to 19 squid 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[64662]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-24T11:47:28.116 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-24T11:47:28.117 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: Upgrade: Setting require_osd_release to 19 squid 2024-03-24T11:47:28.117 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-24T11:47:28.117 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:27 smithi145 ceph-mon[67080]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:47:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:28.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: Upgrade: Setting container_image for all osd 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-24T11:47:28.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: Upgrade: Setting require_osd_release to 19 squid 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-24T11:47:28.228 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:27 smithi149 ceph-mon[58107]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:47:28.954 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[67080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 80/627 objects degraded (12.759%), 23 pgs degraded) 2024-03-24T11:47:28.954 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[67080]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-24T11:47:28.954 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-24T11:47:28.954 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[67080]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[67080]: Upgrade: Setting container_image for all mds 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[64662]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 80/627 objects degraded (12.759%), 23 pgs degraded) 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[64662]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[64662]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[64662]: Upgrade: Setting container_image for all mds 2024-03-24T11:47:28.955 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:28 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:28 smithi149 ceph-mon[58107]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 80/627 objects degraded (12.759%), 23 pgs degraded) 2024-03-24T11:47:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:28 smithi149 ceph-mon[58107]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-24T11:47:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:28 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-24T11:47:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:28 smithi149 ceph-mon[58107]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:47:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:28 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:28 smithi149 ceph-mon[58107]: Upgrade: Setting container_image for all mds 2024-03-24T11:47:29.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:28 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:30.536 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:47:30.845 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[67080]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-24T11:47:30.845 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[67080]: Upgrade: Updating rgw.foo.smithi145.rqljdz (1/4) 2024-03-24T11:47:30.845 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:30.845 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:30.845 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:30.845 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[64662]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-24T11:47:30.846 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[64662]: Upgrade: Updating rgw.foo.smithi145.rqljdz (1/4) 2024-03-24T11:47:30.846 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:30.846 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:30.846 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:30 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:30 smithi149 ceph-mon[58107]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-24T11:47:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:30 smithi149 ceph-mon[58107]: Upgrade: Updating rgw.foo.smithi145.rqljdz (1/4) 2024-03-24T11:47:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:30 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:30 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi145.rqljdz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:30 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:31.188 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:47:31.188 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (7m) 2m ago 12m 25.6M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (6m) 8s ago 11m 53.2M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (6m) 2m ago 11m 55.5M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (6m) 8s ago 16m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (6m) 2m ago 18m 506M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (5m) 2m ago 18m 49.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (5m) 8s ago 17m 40.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (5m) 2m ago 17m 40.4M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (7m) 2m ago 12m 17.5M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (7m) 8s ago 12m 18.8M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (4m) 2m ago 16m 51.2M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (3m) 2m ago 16m 51.7M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:47:31.189 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (3m) 2m ago 15m 41.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (2m) 2m ago 15m 12.5M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (118s) 8s ago 14m 58.6M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 8a8a001df0e1 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (83s) 8s ago 14m 46.7M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 ef3114caac79 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (49s) 8s ago 14m 42.6M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 1223bf8d794a 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (13s) 8s ago 13m 14.2M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 f5d48891e80c 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (6m) 8s ago 13m 69.0M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (11m) 2m ago 11m 72.2M - 17.2.0 e1d6a67b021e 9acfde645093 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (11m) 8s ago 11m 76.6M - 17.2.0 e1d6a67b021e 8d5286781fb0 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (11m) 2m ago 11m 72.8M - 17.2.0 e1d6a67b021e 3a0233b3115f 2024-03-24T11:47:31.190 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (11m) 8s ago 11m 75.7M - 17.2.0 e1d6a67b021e 035e71dc282f 2024-03-24T11:47:31.612 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:47:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:47:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:47:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:47:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:47:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:47:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 13 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:47:31.613 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:47:31.812 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:31 smithi145 ceph-mon[64662]: Deploying daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:47:31.812 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:31 smithi145 ceph-mon[64662]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:31.812 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:31 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1924575260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:31.812 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:31 smithi145 ceph-mon[67080]: Deploying daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:47:31.812 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:31 smithi145 ceph-mon[67080]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:31.812 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:31 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/1924575260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "osd", 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:47:31.959 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "13/23 daemons upgraded", 2024-03-24T11:47:31.960 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading rgw daemons", 2024-03-24T11:47:31.960 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:47:31.960 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:47:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:31 smithi149 ceph-mon[58107]: Deploying daemon rgw.foo.smithi145.rqljdz on smithi145 2024-03-24T11:47:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:31 smithi149 ceph-mon[58107]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:31 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/1924575260' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:47:32.364 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:47:32.812 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[64662]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[64662]: from='client.44488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[64662]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[64662]: from='client.44500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1683099333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[67080]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[67080]: from='client.44488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.813 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[67080]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:32.814 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[67080]: from='client.44500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.814 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:32 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/1683099333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:32 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:32 smithi149 ceph-mon[58107]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:32 smithi149 ceph-mon[58107]: from='client.44488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:32 smithi149 ceph-mon[58107]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:47:32.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:32 smithi149 ceph-mon[58107]: from='client.44500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:32 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/1683099333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:47:33.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:33 smithi145 ceph-mon[67080]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:47:33.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:33 smithi145 ceph-mon[64662]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:47:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:33 smithi149 ceph-mon[58107]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:47:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:35.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:35.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:35.704 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:47:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:47:35] "GET /metrics HTTP/1.1" 200 38562 "" "Prometheus/2.33.4" 2024-03-24T11:47:36.765 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[64662]: Upgrade: Updating rgw.smpl.smithi145.wrtpsu (2/4) 2024-03-24T11:47:36.765 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:36.765 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[64662]: Deploying daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[64662]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[67080]: Upgrade: Updating rgw.smpl.smithi145.wrtpsu (2/4) 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[67080]: Deploying daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:47:36.766 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:36 smithi145 ceph-mon[67080]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:47:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:36 smithi149 ceph-mon[58107]: Upgrade: Updating rgw.smpl.smithi145.wrtpsu (2/4) 2024-03-24T11:47:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi145.wrtpsu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:36 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:36 smithi149 ceph-mon[58107]: Deploying daemon rgw.smpl.smithi145.wrtpsu on smithi145 2024-03-24T11:47:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:36 smithi149 ceph-mon[58107]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:47:38.797 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:38 smithi145 ceph-mon[64662]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 102 B/s wr, 101 op/s 2024-03-24T11:47:38.797 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:38 smithi145 ceph-mon[67080]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 102 B/s wr, 101 op/s 2024-03-24T11:47:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:38 smithi149 ceph-mon[58107]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 102 B/s wr, 101 op/s 2024-03-24T11:47:39.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:39 smithi145 ceph-mon[67080]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 88 B/s wr, 87 op/s 2024-03-24T11:47:39.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:39.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:39 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:39 smithi145 ceph-mon[64662]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 88 B/s wr, 87 op/s 2024-03-24T11:47:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:39.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:39 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:39 smithi149 ceph-mon[58107]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 88 B/s wr, 87 op/s 2024-03-24T11:47:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:40.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:39 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:42 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:42 smithi149 ceph-mon[58107]: Upgrade: Updating rgw.foo.smithi149.gflbhe (3/4) 2024-03-24T11:47:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:42.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:42 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:42.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:42 smithi149 ceph-mon[58107]: Deploying daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:47:42.726 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:42 smithi149 ceph-mon[58107]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[67080]: Upgrade: Updating rgw.foo.smithi149.gflbhe (3/4) 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[67080]: Deploying daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[67080]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:47:42.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:42.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[64662]: Upgrade: Updating rgw.foo.smithi149.gflbhe (3/4) 2024-03-24T11:47:42.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:42.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.gflbhe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:42.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:42.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[64662]: Deploying daemon rgw.foo.smithi149.gflbhe on smithi149 2024-03-24T11:47:42.863 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:42 smithi145 ceph-mon[64662]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:47:44.912 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:44 smithi149 ceph-mon[58107]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:45.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:44 smithi145 ceph-mon[67080]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:45.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:44 smithi145 ceph-mon[64662]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:45.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:47:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:47:45] "GET /metrics HTTP/1.1" 200 38562 "" "Prometheus/2.33.4" 2024-03-24T11:47:46.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:46 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:46.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:46 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:46.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:46 smithi145 ceph-mon[67080]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:46 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:46 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:46 smithi145 ceph-mon[64662]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:46.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:46 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:46.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:46 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:46.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:46 smithi149 ceph-mon[58107]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:48.221 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:47 smithi149 ceph-mon[58107]: Upgrade: Updating rgw.smpl.smithi149.djutzp (4/4) 2024-03-24T11:47:48.221 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:48.221 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:48.221 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:47 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:48.221 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:47 smithi149 ceph-mon[58107]: Deploying daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:47:48.221 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:47 smithi149 ceph-mon[58107]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-03-24T11:47:48.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[64662]: Upgrade: Updating rgw.smpl.smithi149.djutzp (4/4) 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[64662]: Deploying daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[64662]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[67080]: Upgrade: Updating rgw.smpl.smithi149.djutzp (4/4) 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.djutzp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:47:48.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:48.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[67080]: Deploying daemon rgw.smpl.smithi149.djutzp on smithi149 2024-03-24T11:47:48.363 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:47 smithi145 ceph-mon[67080]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-03-24T11:47:49.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:49 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:50.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:49 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:50.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:49 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:47:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:50 smithi149 ceph-mon[58107]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:51.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:50 smithi145 ceph-mon[67080]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:51.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:50 smithi145 ceph-mon[64662]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:52.259 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:52 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:52.259 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:52.259 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:52 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:52.259 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:52 smithi149 ceph-mon[58107]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:52.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:52.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:52.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:52.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[64662]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:52.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:47:52.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:52.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:52.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:52 smithi145 ceph-mon[67080]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:53.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:53 smithi149 ceph-mon[58107]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 255 B/s wr, 252 op/s 2024-03-24T11:47:54.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:53 smithi145 ceph-mon[64662]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 255 B/s wr, 252 op/s 2024-03-24T11:47:54.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:53 smithi145 ceph-mon[67080]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 255 B/s wr, 252 op/s 2024-03-24T11:47:55.845 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:47:55] "GET /metrics HTTP/1.1" 200 38562 "" "Prometheus/2.33.4" 2024-03-24T11:47:56.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[64662]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:55 smithi145 ceph-mon[67080]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:55 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:55 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:55 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:55 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:56.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:55 smithi149 ceph-mon[58107]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:57.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:56 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:57.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:56 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:47:58.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:47:57 smithi145 ceph-mon[64662]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:58.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:47:57 smithi145 ceph-mon[67080]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:47:58.131 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:47:57 smithi149 ceph-mon[58107]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: Detected new or changed devices on smithi145 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: Detected new or changed devices on smithi149 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:00.676 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi145.rqljdz"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi145.rqljdz"}]': finished 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.gflbhe"}]: dispatch 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.gflbhe"}]': finished 2024-03-24T11:48:00.677 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi145.wrtpsu"}]: dispatch 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi145.wrtpsu"}]': finished 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.djutzp"}]: dispatch 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.djutzp"}]': finished 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.678 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: Detected new or changed devices on smithi145 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: Detected new or changed devices on smithi149 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.679 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi145.rqljdz"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi145.rqljdz"}]': finished 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.gflbhe"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.gflbhe"}]': finished 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi145.wrtpsu"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi145.wrtpsu"}]': finished 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.djutzp"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.djutzp"}]': finished 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.680 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.681 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:00 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.877 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: Detected new or changed devices on smithi145 2024-03-24T11:48:00.877 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.877 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.877 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: Detected new or changed devices on smithi149 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi145.rqljdz"}]: dispatch 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi145.rqljdz"}]': finished 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.gflbhe"}]: dispatch 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.gflbhe"}]': finished 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi145.wrtpsu"}]: dispatch 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi145.wrtpsu"}]': finished 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.djutzp"}]: dispatch 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.djutzp"}]': finished 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:00.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:00 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:01.917 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[64662]: Upgrade: Setting container_image for all rgw 2024-03-24T11:48:01.917 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[64662]: Upgrade: Setting container_image for all rbd-mirror 2024-03-24T11:48:01.917 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[64662]: Upgrade: Setting container_image for all ceph-exporter 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[67080]: Upgrade: Setting container_image for all rgw 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[67080]: Upgrade: Setting container_image for all rbd-mirror 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[67080]: Upgrade: Setting container_image for all ceph-exporter 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:48:01.918 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:01 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:01 smithi149 ceph-mon[58107]: Upgrade: Setting container_image for all rgw 2024-03-24T11:48:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:01 smithi149 ceph-mon[58107]: Upgrade: Setting container_image for all rbd-mirror 2024-03-24T11:48:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:01 smithi149 ceph-mon[58107]: Upgrade: Setting container_image for all ceph-exporter 2024-03-24T11:48:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:01 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:01 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi145.hnqhcg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:48:01.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:01 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:02.717 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:48:02.861 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:02.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[64662]: Upgrade: Updating iscsi.foo.smithi145.hnqhcg 2024-03-24T11:48:02.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[64662]: Deploying daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:48:02.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[64662]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:48:02.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:02.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[67080]: Upgrade: Updating iscsi.foo.smithi145.hnqhcg 2024-03-24T11:48:02.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[67080]: Deploying daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:48:02.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:02 smithi145 ceph-mon[67080]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:48:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:02 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:02 smithi149 ceph-mon[58107]: Upgrade: Updating iscsi.foo.smithi145.hnqhcg 2024-03-24T11:48:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:02 smithi149 ceph-mon[58107]: Deploying daemon iscsi.foo.smithi145.hnqhcg on smithi145 2024-03-24T11:48:02.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:02 smithi149 ceph-mon[58107]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (7m) 8s ago 12m 25.6M - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (7m) 8s ago 12m 52.1M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 running (6m) 8s ago 11m 66.3M - 3.5 e1d6a67b021e 461e73f32296 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (6m) 8s ago 17m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (7m) 8s ago 18m 512M - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (6m) 8s ago 19m 57.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (5m) 8s ago 17m 42.5M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:48:03.447 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (5m) 8s ago 17m 45.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (8m) 8s ago 13m 19.1M - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (7m) 8s ago 13m 18.6M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (4m) 8s ago 17m 62.4M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (4m) 8s ago 16m 65.0M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (3m) 8s ago 16m 49.8M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (3m) 8s ago 15m 62.1M 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (2m) 8s ago 15m 65.4M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 8a8a001df0e1 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (116s) 8s ago 15m 50.3M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 ef3114caac79 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (81s) 8s ago 14m 47.7M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 1223bf8d794a 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (46s) 8s ago 14m 48.4M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 f5d48891e80c 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (6m) 8s ago 13m 69.0M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (30s) 8s ago 12m 77.4M - 19.0.0-1667-gdb0330b1 4f325f8440e9 fa32ee235bb4 2024-03-24T11:48:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (18s) 8s ago 12m 74.8M - 19.0.0-1667-gdb0330b1 4f325f8440e9 e412422eaed7 2024-03-24T11:48:03.449 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (24s) 8s ago 11m 74.8M - 19.0.0-1667-gdb0330b1 4f325f8440e9 8b6fd7ccf24e 2024-03-24T11:48:03.449 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (12s) 8s ago 11m 73.9M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6530a65bc18b 2024-03-24T11:48:03.844 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:48:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 17 2024-03-24T11:48:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:48:03.846 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:48:04.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[67080]: from='client.44596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[67080]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[67080]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[67080]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-03-24T11:48:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[64662]: from='client.44596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[64662]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[64662]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:03 smithi145 ceph-mon[64662]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw", 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "osd", 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "17/23 daemons upgraded", 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Currently upgrading iscsi daemons", 2024-03-24T11:48:04.173 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": false 2024-03-24T11:48:04.174 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:48:04.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:03 smithi149 ceph-mon[58107]: from='client.44596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:03 smithi149 ceph-mon[58107]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:03 smithi149 ceph-mon[58107]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:03 smithi149 ceph-mon[58107]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 85 op/s 2024-03-24T11:48:04.560 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_OK 2024-03-24T11:48:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/806256721' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[64662]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3916261803' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:48:04.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/806256721' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[67080]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.862 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3916261803' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:48:04.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:04 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:05.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:04 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/806256721' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:05.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:04 smithi149 ceph-mon[58107]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:05.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:04 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3916261803' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:48:05.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:04 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:05.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:05 smithi145 ceph-mon[67080]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:05.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:05 smithi145 ceph-mon[64662]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:05.862 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:48:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:48:05] "GET /metrics HTTP/1.1" 200 38634 "" "Prometheus/2.33.4" 2024-03-24T11:48:06.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:05 smithi149 ceph-mon[58107]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:08.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:07 smithi145 ceph-mon[67080]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:08.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:07 smithi145 ceph-mon[64662]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:08.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:07 smithi149 ceph-mon[58107]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:10.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:09 smithi145 ceph-mon[64662]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:10.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:09 smithi145 ceph-mon[67080]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:10.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:09 smithi149 ceph-mon[58107]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:12.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:11 smithi145 ceph-mon[64662]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:12.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:11 smithi145 ceph-mon[64662]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:11 smithi145 ceph-mon[67080]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:12.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:11 smithi145 ceph-mon[67080]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:12.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:11 smithi149 ceph-mon[58107]: from='client.25118 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:12.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:11 smithi149 ceph-mon[58107]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:14.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:13 smithi145 ceph-mon[64662]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:14.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:13 smithi145 ceph-mon[67080]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:14.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:13 smithi149 ceph-mon[58107]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:15.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:48:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:48:14.845+0000 7f4fcba5d700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:48:15.772 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:48:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:48:15] "GET /metrics HTTP/1.1" 200 38634 "" "Prometheus/2.33.4" 2024-03-24T11:48:16.059 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:15 smithi145 ceph-mon[67080]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:48:16.059 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:15 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:16.059 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:15 smithi145 ceph-mon[67080]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-03-24T11:48:16.059 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:15 smithi145 ceph-mon[64662]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:48:16.059 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:15 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:16.060 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:15 smithi145 ceph-mon[64662]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-03-24T11:48:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:15 smithi149 ceph-mon[58107]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:48:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:15 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:16.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:15 smithi149 ceph-mon[58107]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-03-24T11:48:16.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:16 smithi149 ceph-mon[58107]: Health check failed: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-24T11:48:17.280 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:16 smithi145 ceph-mon[64662]: Health check failed: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-24T11:48:17.281 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:16 smithi145 ceph-mon[67080]: Health check failed: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-24T11:48:18.165 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:17 smithi149 ceph-mon[58107]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:18.207 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:17 smithi145 ceph-mon[64662]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:18.207 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:17 smithi145 ceph-mon[67080]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:19.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:19.863 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:19.976 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:20 smithi149 ceph-mon[58107]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-03-24T11:48:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:20 smithi149 ceph-mon[58107]: Checking dashboard <-> RGW credentials 2024-03-24T11:48:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:20 smithi149 ceph-mon[58107]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-03-24T11:48:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:20 smithi149 ceph-mon[58107]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-03-24T11:48:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:20 smithi149 ceph-mon[58107]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[64662]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[64662]: Checking dashboard <-> RGW credentials 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[64662]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[64662]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[64662]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[67080]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[67080]: Checking dashboard <-> RGW credentials 2024-03-24T11:48:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[67080]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-03-24T11:48:21.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[67080]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-03-24T11:48:21.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:21.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:21.113 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:20 smithi145 ceph-mon[67080]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:48:22.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:22 smithi149 ceph-mon[58107]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 273 B/s rd, 0 op/s 2024-03-24T11:48:23.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:22 smithi145 ceph-mon[64662]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 273 B/s rd, 0 op/s 2024-03-24T11:48:23.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:22 smithi145 ceph-mon[67080]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 273 B/s rd, 0 op/s 2024-03-24T11:48:24.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:24 smithi149 ceph-mon[58107]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-03-24T11:48:25.102 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:24 smithi145 ceph-mon[64662]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-03-24T11:48:25.102 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:24 smithi145 ceph-mon[67080]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-03-24T11:48:25.750 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:48:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:48:25] "GET /metrics HTTP/1.1" 200 38713 "" "Prometheus/2.33.4" 2024-03-24T11:48:26.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:26 smithi149 ceph-mon[58107]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-24T11:48:26.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:26 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/4266115912' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:48:26.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:26 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/948327092' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]: dispatch 2024-03-24T11:48:26.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:26 smithi149 ceph-mon[58107]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]: dispatch 2024-03-24T11:48:27.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[67080]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-24T11:48:27.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/4266115912' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:48:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/948327092' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]: dispatch 2024-03-24T11:48:27.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[67080]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]: dispatch 2024-03-24T11:48:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[64662]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-24T11:48:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/4266115912' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:48:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/948327092' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]: dispatch 2024-03-24T11:48:27.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:26 smithi145 ceph-mon[64662]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]: dispatch 2024-03-24T11:48:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:27 smithi149 ceph-mon[58107]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]': finished 2024-03-24T11:48:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:27 smithi149 ceph-mon[58107]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:48:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:27 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2002467438' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/636932421"}]: dispatch 2024-03-24T11:48:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:27 smithi145 ceph-mon[64662]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]': finished 2024-03-24T11:48:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:27 smithi145 ceph-mon[64662]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:48:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:27 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2002467438' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/636932421"}]: dispatch 2024-03-24T11:48:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:27 smithi145 ceph-mon[67080]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/2573589722"}]': finished 2024-03-24T11:48:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:27 smithi145 ceph-mon[67080]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:48:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:27 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2002467438' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/636932421"}]: dispatch 2024-03-24T11:48:28.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:28 smithi149 ceph-mon[58107]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-24T11:48:28.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:28 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2002467438' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/636932421"}]': finished 2024-03-24T11:48:28.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:28 smithi149 ceph-mon[58107]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:48:28.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:28 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3941726353' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1519562264"}]: dispatch 2024-03-24T11:48:29.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[64662]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-24T11:48:29.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2002467438' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/636932421"}]': finished 2024-03-24T11:48:29.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[64662]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:48:29.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3941726353' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1519562264"}]: dispatch 2024-03-24T11:48:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[67080]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-24T11:48:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2002467438' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/636932421"}]': finished 2024-03-24T11:48:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[67080]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:48:29.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:28 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3941726353' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1519562264"}]: dispatch 2024-03-24T11:48:29.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:29 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3941726353' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1519562264"}]': finished 2024-03-24T11:48:29.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:29 smithi149 ceph-mon[58107]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:48:29.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:29 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3101109193' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3625648990"}]: dispatch 2024-03-24T11:48:30.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:29 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3941726353' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1519562264"}]': finished 2024-03-24T11:48:30.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:29 smithi145 ceph-mon[64662]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:48:30.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:29 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3101109193' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3625648990"}]: dispatch 2024-03-24T11:48:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:29 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3941726353' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/1519562264"}]': finished 2024-03-24T11:48:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:29 smithi145 ceph-mon[67080]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:48:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:29 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3101109193' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3625648990"}]: dispatch 2024-03-24T11:48:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:30 smithi149 ceph-mon[58107]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:48:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:30 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3101109193' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3625648990"}]': finished 2024-03-24T11:48:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:30 smithi149 ceph-mon[58107]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:48:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:30 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2705842466' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]: dispatch 2024-03-24T11:48:30.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:30 smithi149 ceph-mon[58107]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]: dispatch 2024-03-24T11:48:31.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[67080]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3101109193' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3625648990"}]': finished 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[67080]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2705842466' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]: dispatch 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[67080]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]: dispatch 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[64662]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3101109193' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:0/3625648990"}]': finished 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[64662]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2705842466' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]: dispatch 2024-03-24T11:48:31.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:30 smithi145 ceph-mon[64662]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]: dispatch 2024-03-24T11:48:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:31 smithi149 ceph-mon[58107]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]': finished 2024-03-24T11:48:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:31 smithi149 ceph-mon[58107]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:48:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:31 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3903698003' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3222839647"}]: dispatch 2024-03-24T11:48:32.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:31 smithi145 ceph-mon[67080]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]': finished 2024-03-24T11:48:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:31 smithi145 ceph-mon[67080]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:48:32.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:31 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3903698003' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3222839647"}]: dispatch 2024-03-24T11:48:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:31 smithi145 ceph-mon[64662]: from='client.? ' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6801/3222839647"}]': finished 2024-03-24T11:48:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:31 smithi145 ceph-mon[64662]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:48:32.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:31 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3903698003' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3222839647"}]: dispatch 2024-03-24T11:48:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:32 smithi149 ceph-mon[58107]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:32 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3903698003' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3222839647"}]': finished 2024-03-24T11:48:32.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:32 smithi149 ceph-mon[58107]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:48:33.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:32 smithi145 ceph-mon[67080]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:33.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:32 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3903698003' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3222839647"}]': finished 2024-03-24T11:48:33.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:32 smithi145 ceph-mon[67080]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:48:33.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:32 smithi145 ceph-mon[64662]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 276 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:48:33.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:32 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3903698003' entity='client.iscsi.foo.smithi145.hnqhcg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.145:6800/3222839647"}]': finished 2024-03-24T11:48:33.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:32 smithi145 ceph-mon[64662]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:48:34.929 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:48:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:34 smithi149 ceph-mon[58107]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-03-24T11:48:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:35.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:34 smithi145 ceph-mon[64662]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-03-24T11:48:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:35.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:35.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:34 smithi145 ceph-mon[67080]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-03-24T11:48:35.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:48:35.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:35.261 INFO:teuthology.orchestra.run.smithi145.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed." 2024-03-24T11:48:35.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:48:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:48:35] "GET /metrics HTTP/1.1" 200 38711 "" "Prometheus/2.33.4" 2024-03-24T11:48:36.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch ps' 2024-03-24T11:48:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:36 smithi149 ceph-mon[58107]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:36 smithi149 ceph-mon[58107]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:48:36.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:36 smithi149 ceph-mon[58107]: from='client.44731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:37.109 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:36 smithi145 ceph-mon[64662]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:37.109 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:36 smithi145 ceph-mon[64662]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:48:37.109 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:36 smithi145 ceph-mon[64662]: from='client.44731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:37.109 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:36 smithi145 ceph-mon[67080]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:37.109 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:36 smithi145 ceph-mon[67080]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:48:37.109 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:36 smithi145 ceph-mon[67080]: from='client.44731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:37 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:38.109 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:48:38.109 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (8m) 19s ago 13m - - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (7m) 19s ago 13m 52.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 unknown 19s ago 12m - - 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (7m) 19s ago 17m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (7m) 19s ago 19m - - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (6m) 19s ago 19m - 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (6m) 19s ago 18m 43.8M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (6m) 19s ago 18m - 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (8m) 19s ago 14m - - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (8m) 19s ago 14m 18.4M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (5m) 19s ago 17m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (4m) 19s ago 17m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:48:38.110 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (4m) 19s ago 16m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (3m) 19s ago 16m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (3m) 19s ago 16m 64.5M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 8a8a001df0e1 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (2m) 19s ago 15m 50.5M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 ef3114caac79 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (115s) 19s ago 15m 46.9M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 1223bf8d794a 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (80s) 19s ago 14m 47.6M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 f5d48891e80c 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (7m) 19s ago 14m 69.0M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (64s) 19s ago 12m - - 19.0.0-1667-gdb0330b1 4f325f8440e9 fa32ee235bb4 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (53s) 19s ago 12m 75.1M - 19.0.0-1667-gdb0330b1 4f325f8440e9 e412422eaed7 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (58s) 19s ago 12m - - 19.0.0-1667-gdb0330b1 4f325f8440e9 8b6fd7ccf24e 2024-03-24T11:48:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (47s) 19s ago 12m 74.2M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6530a65bc18b 2024-03-24T11:48:38.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:37 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:38.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:37 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:38.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions' 2024-03-24T11:48:38.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:38 smithi145 ceph-mon[67080]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:38.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:38 smithi145 ceph-mon[64662]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:38.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:38 smithi149 ceph-mon[58107]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:39 smithi149 ceph-mon[58107]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:40.108 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:39 smithi145 ceph-mon[67080]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:40.109 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:39 smithi145 ceph-mon[64662]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:48:40.901 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:48:40.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:48:40.902 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:48:40.902 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 17 2024-03-24T11:48:40.902 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:48:40.902 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:48:40.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:40 smithi149 ceph-mon[58107]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:41.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:40 smithi145 ceph-mon[67080]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:41.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:40 smithi145 ceph-mon[64662]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:48:41.599 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-24T11:48:41.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:41 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/946169771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:41.861 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:41 smithi145 ceph-mon[67080]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:48:41.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:41 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/946169771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:41.862 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:41 smithi145 ceph-mon[64662]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:48:41.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:41 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/946169771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:48:41.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:41 smithi149 ceph-mon[58107]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:48:43.300 INFO:teuthology.orchestra.run.smithi145.stdout:wait for servicemap items w/ changing names to refresh 2024-03-24T11:48:44.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'sleep 60' 2024-03-24T11:48:44.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:44 smithi149 ceph-mon[58107]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:44.572 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:44 smithi145 ceph-mon[67080]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:44.572 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:44 smithi145 ceph-mon[64662]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:45.824 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:48:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:48:45] "GET /metrics HTTP/1.1" 200 38711 "" "Prometheus/2.33.4" 2024-03-24T11:48:46.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:46 smithi149 ceph-mon[58107]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:46.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:46 smithi145 ceph-mon[64662]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:46.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:46 smithi145 ceph-mon[67080]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:47 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:47.525 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:47 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:47.525 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:47 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:48.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:48 smithi145 ceph-mon[64662]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:48.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:48 smithi145 ceph-mon[67080]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:48.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:48 smithi149 ceph-mon[58107]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:50.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:50 smithi145 ceph-mon[64662]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:50.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:50.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:50 smithi145 ceph-mon[67080]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:50.612 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:50 smithi149 ceph-mon[58107]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:50.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:48:52.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:52 smithi145 ceph-mon[67080]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:52.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:52 smithi145 ceph-mon[64662]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:52.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:52 smithi149 ceph-mon[58107]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:54.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:54 smithi145 ceph-mon[67080]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:54.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:54 smithi145 ceph-mon[64662]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:54.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:54 smithi149 ceph-mon[58107]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:55.861 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:48:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:48:55] "GET /metrics HTTP/1.1" 200 38708 "" "Prometheus/2.33.4" 2024-03-24T11:48:56.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:56 smithi145 ceph-mon[67080]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:56.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:56 smithi145 ceph-mon[64662]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:56.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:56 smithi149 ceph-mon[58107]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:48:57.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:57 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:57.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:57 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:57.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:57 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:48:58.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:48:58 smithi145 ceph-mon[67080]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:58.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:48:58 smithi145 ceph-mon[64662]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:48:58.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:48:58 smithi149 ceph-mon[58107]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:00.527 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:00 smithi149 ceph-mon[58107]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:00.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:00 smithi145 ceph-mon[67080]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:00.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:00 smithi145 ceph-mon[64662]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:02.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:02 smithi145 ceph-mon[67080]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:02.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:02 smithi145 ceph-mon[64662]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:02.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:02 smithi149 ceph-mon[58107]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:04.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:04 smithi145 ceph-mon[67080]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:04.612 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:04 smithi145 ceph-mon[64662]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:04.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:04 smithi149 ceph-mon[58107]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:05.782 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:05 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:05.783 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:49:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:49:05] "GET /metrics HTTP/1.1" 200 38711 "" "Prometheus/2.33.4" 2024-03-24T11:49:05.783 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:05 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:05.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:05 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:06.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:06 smithi149 ceph-mon[58107]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:07.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:06 smithi145 ceph-mon[64662]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:07.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:06 smithi145 ceph-mon[67080]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:07.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:07 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:08.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:07 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:08.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:07 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:08.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:08 smithi149 ceph-mon[58107]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:09.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:08 smithi145 ceph-mon[64662]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:09.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:08 smithi145 ceph-mon[67080]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:10.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:10 smithi149 ceph-mon[58107]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:11.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:10 smithi145 ceph-mon[64662]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:11.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:10 smithi145 ceph-mon[67080]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:12.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:12 smithi149 ceph-mon[58107]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:13.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:12 smithi145 ceph-mon[64662]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:13.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:12 smithi145 ceph-mon[67080]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:14.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:14 smithi149 ceph-mon[58107]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:15.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:14 smithi145 ceph-mon[67080]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:15.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:14 smithi145 ceph-mon[64662]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:15.792 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:49:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:49:15] "GET /metrics HTTP/1.1" 200 38711 "" "Prometheus/2.33.4" 2024-03-24T11:49:16.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:16 smithi149 ceph-mon[58107]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:16 smithi145 ceph-mon[67080]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:17.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:16 smithi145 ceph-mon[64662]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:17.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:17 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:18.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:17 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:18.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:17 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:18.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:18 smithi149 ceph-mon[58107]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:19.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:18 smithi145 ceph-mon[67080]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:19.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:18 smithi145 ceph-mon[64662]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:19.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:19 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:20.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:19 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:20.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:19 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:20 smithi149 ceph-mon[58107]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:20.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:20 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:49:21.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[64662]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:21.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:21.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:49:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[67080]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:21.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:20 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' 2024-03-24T11:49:21.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:21 smithi149 ceph-mon[58107]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-03-24T11:49:21.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:21 smithi149 ceph-mon[58107]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:22.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:21 smithi145 ceph-mon[64662]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-03-24T11:49:22.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:21 smithi145 ceph-mon[64662]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:22.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:21 smithi145 ceph-mon[67080]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-03-24T11:49:22.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:21 smithi145 ceph-mon[67080]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:23.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:23 smithi149 ceph-mon[58107]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:24.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:23 smithi145 ceph-mon[64662]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:24.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:23 smithi145 ceph-mon[67080]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:25.802 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:25 smithi145 ceph-mon[64662]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:25.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:49:25 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:49:25] "GET /metrics HTTP/1.1" 200 38713 "" "Prometheus/2.33.4" 2024-03-24T11:49:25.803 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:25 smithi145 ceph-mon[67080]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:25.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:25 smithi149 ceph-mon[58107]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:27 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:27.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:27 smithi149 ceph-mon[58107]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-24T11:49:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:27 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:28.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:27 smithi145 ceph-mon[64662]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-24T11:49:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:27 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:28.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:27 smithi145 ceph-mon[67080]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-24T11:49:29.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:29 smithi149 ceph-mon[58107]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:30.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:29 smithi145 ceph-mon[64662]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:30.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:29 smithi145 ceph-mon[67080]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:49:31.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:31 smithi149 ceph-mon[58107]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:49:32.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:31 smithi145 ceph-mon[64662]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:49:32.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:31 smithi145 ceph-mon[67080]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:49:33.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:33 smithi149 ceph-mon[58107]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:34.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:33 smithi145 ceph-mon[64662]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:34.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:33 smithi145 ceph-mon[67080]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:34.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:34 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:35.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:34 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:35.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:34 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:35.813 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:35 smithi145 ceph-mon[64662]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:35.813 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:35 smithi145 ceph-mon[67080]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:35.813 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:49:35 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:49:35] "GET /metrics HTTP/1.1" 200 38712 "" "Prometheus/2.33.4" 2024-03-24T11:49:35.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:35 smithi149 ceph-mon[58107]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:37 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:37.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:37 smithi149 ceph-mon[58107]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:38.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:37 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:38.111 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:37 smithi145 ceph-mon[64662]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:38.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:37 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:38.112 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:37 smithi145 ceph-mon[67080]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:39.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:39 smithi149 ceph-mon[58107]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:40.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:39 smithi145 ceph-mon[67080]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:40.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:39 smithi145 ceph-mon[64662]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:41.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:41 smithi149 ceph-mon[58107]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:42.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:41 smithi145 ceph-mon[67080]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:42.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:41 smithi145 ceph-mon[64662]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:43.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:42 smithi145 ceph-mon[67080]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:43.112 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:42 smithi145 ceph-mon[64662]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:43.225 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:42 smithi149 ceph-mon[58107]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:45.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:45 smithi145 ceph-mon[67080]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:45.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:45 smithi145 ceph-mon[64662]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:45.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:45 smithi149 ceph-mon[58107]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-24T11:49:45.823 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:49:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:49:45] "GET /metrics HTTP/1.1" 200 38712 "" "Prometheus/2.33.4" 2024-03-24T11:49:46.536 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch ps' 2024-03-24T11:49:47.104 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:47 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:47.104 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:47 smithi145 ceph-mon[64662]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:47.105 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:47 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:47.105 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:47 smithi145 ceph-mon[67080]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:47 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:47.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:47 smithi149 ceph-mon[58107]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:48.711 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:alertmanager.a smithi145 *:9093,9094 running (9m) 89s ago 14m - - 0.23.0 ba2b418f427c e32a8e22dad4 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:grafana.a smithi149 *:3000 running (9m) 90s ago 14m 52.6M - 8.3.5 dad864ee21e9 1465ddad3f0c 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:iscsi.foo.smithi145.hnqhcg smithi145 unknown 89s ago 13m - - 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi149 *:8443,9283,8765 running (8m) 90s ago 19m 452M - 19.0.0-1667-gdb0330b1 4f325f8440e9 bba2809eb6be 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi145 *:8443,9283,8765 running (8m) 89s ago 20m - - 19.0.0-1667-gdb0330b1 4f325f8440e9 387fa4d011fe 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (8m) 89s ago 20m - 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 7ad6536c6cf9 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi149 running (7m) 90s ago 19m 43.8M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 6dedc9190b9e 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi145 running (7m) 89s ago 19m - 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 4c983fe930c0 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.a smithi145 *:9100 running (9m) 89s ago 15m - - 1.3.1 1dbe0e931976 77953b6c5fa5 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:node-exporter.b smithi149 *:9100 running (9m) 90s ago 15m 18.4M - 1.3.1 1dbe0e931976 54579c2f54d5 2024-03-24T11:49:48.712 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (6m) 89s ago 18m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5960e0232af 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi145 running (6m) 89s ago 18m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 848b03020bb1 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi145 running (5m) 89s ago 18m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 a91a34aa8465 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (4m) 89s ago 17m - 2736M 19.0.0-1667-gdb0330b1 4f325f8440e9 ffbb9bef1b39 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi149 running (4m) 90s ago 17m 64.5M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 8a8a001df0e1 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi149 running (3m) 90s ago 16m 50.5M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 ef3114caac79 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi149 running (3m) 90s ago 16m 46.9M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 1223bf8d794a 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi149 running (2m) 90s ago 16m 47.6M 1454M 19.0.0-1667-gdb0330b1 4f325f8440e9 f5d48891e80c 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:prometheus.a smithi149 *:9095 running (8m) 90s ago 15m 69.0M - 2.33.4 514e6a882f6e 595718fc2704 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi145.rqljdz smithi145 *:8000 running (2m) 89s ago 13m - - 19.0.0-1667-gdb0330b1 4f325f8440e9 fa32ee235bb4 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.foo.smithi149.gflbhe smithi149 *:8000 running (2m) 90s ago 13m 75.1M - 19.0.0-1667-gdb0330b1 4f325f8440e9 e412422eaed7 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi145.wrtpsu smithi145 *:80 running (2m) 89s ago 13m - - 19.0.0-1667-gdb0330b1 4f325f8440e9 8b6fd7ccf24e 2024-03-24T11:49:48.713 INFO:teuthology.orchestra.run.smithi145.stdout:rgw.smpl.smithi149.djutzp smithi149 *:80 running (118s) 90s ago 13m 74.2M - 19.0.0-1667-gdb0330b1 4f325f8440e9 6530a65bc18b 2024-03-24T11:49:49.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:49 smithi145 ceph-mon[64662]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:49.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:49 smithi145 ceph-mon[64662]: from='client.44749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:49.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:49 smithi145 ceph-mon[67080]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:49.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:49 smithi145 ceph-mon[67080]: from='client.44749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:49.403 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions' 2024-03-24T11:49:49.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:49 smithi149 ceph-mon[58107]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:49.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:49 smithi149 ceph-mon[58107]: from='client.44749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:50.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:50 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:50.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:50 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:50.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:50 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:51.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:51 smithi145 ceph-mon[64662]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:51.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:51 smithi145 ceph-mon[67080]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:51.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:51 smithi149 ceph-mon[58107]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:51.549 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: "mon": { 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 3 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr": { 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 2 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: "osd": { 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 8 2024-03-24T11:49:51.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:49:51.551 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw": { 2024-03-24T11:49:51.551 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 4 2024-03-24T11:49:51.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-24T11:49:51.551 INFO:teuthology.orchestra.run.smithi145.stdout: "overall": { 2024-03-24T11:49:51.551 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)": 17 2024-03-24T11:49:51.551 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-24T11:49:51.551 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:49:52.145 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:52 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/1967142971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:49:52.145 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:52 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/1967142971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:49:52.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch upgrade status' 2024-03-24T11:49:52.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:52 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/1967142971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:49:53.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:53 smithi145 ceph-mon[67080]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:53.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:53 smithi145 ceph-mon[64662]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:53.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:53 smithi149 ceph-mon[58107]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:54.132 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "in_progress": true, 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "services_complete": [ 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "rgw", 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr", 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "osd", 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "mon" 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "progress": "17/23 daemons upgraded", 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed.", 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "is_paused": true 2024-03-24T11:49:54.133 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:49:54.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph health detail' 2024-03-24T11:49:55.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:55 smithi145 ceph-mon[64662]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:55.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:55 smithi145 ceph-mon[64662]: from='client.44761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:55.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:55 smithi145 ceph-mon[67080]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:55.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:55 smithi145 ceph-mon[67080]: from='client.44761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:55.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:55 smithi149 ceph-mon[58107]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:55.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:55 smithi149 ceph-mon[58107]: from='client.44761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:55.834 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:49:55 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:49:55] "GET /metrics HTTP/1.1" 200 38713 "" "Prometheus/2.33.4" 2024-03-24T11:49:57.049 INFO:teuthology.orchestra.run.smithi145.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:49:57.049 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:49:57.049 INFO:teuthology.orchestra.run.smithi145.stdout: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrade daemon: iscsi.foo.smithi145.hnqhcg: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:Creating ceph-iscsi config... 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2da2bb64be\x2de9d1\x2d11ee\x2d95cd\x2d87774f69a715.slice/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout:Traceback (most recent call last): 2024-03-24T11:49:57.050 INFO:teuthology.orchestra.run.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: return _run_code(code, main_globals, None, 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: exec(code, run_globals) 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1029, in deploy_daemon 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1165, in deploy_daemon_units 2024-03-24T11:49:57.051 INFO:teuthology.orchestra.run.smithi145.stdout: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:49:57.052 INFO:teuthology.orchestra.run.smithi145.stdout:RuntimeError: Failed command: systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:49:57.052 INFO:teuthology.orchestra.run.smithi145.stdout:See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:49:57.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:57 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:57.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:57 smithi145 ceph-mon[64662]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:57.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:57 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/3843350511' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:49:57.361 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:57 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:57.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:57 smithi145 ceph-mon[67080]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:57.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:57 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/3843350511' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:49:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:57 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:49:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:57 smithi149 ceph-mon[58107]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:49:57.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:57 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/3843350511' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-24T11:49:57.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-24T11:49:59.168 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:49:59 smithi145 ceph-mon[67080]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:59.169 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:49:59 smithi145 ceph-mon[64662]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:59.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:49:59 smithi149 ceph-mon[58107]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:49:59.735 INFO:teuthology.orchestra.run.smithi145.stdout:true 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/295854291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Upgrade daemon: iscsi.foo.smithi145.hnqhcg: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Creating ceph-iscsi config... 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2da2bb64be\x2de9d1\x2d11ee\x2d95cd\x2d87774f69a715.slice/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service 2024-03-24T11:50:00.362 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Non-zero exit code 1 from systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: Traceback (most recent call last): 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: return _run_code(code, main_globals, None, 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: exec(code, run_globals) 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:50:00.363 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1029, in deploy_daemon 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/295854291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Upgrade daemon: iscsi.foo.smithi145.hnqhcg: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Creating ceph-iscsi config... 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2da2bb64be\x2de9d1\x2d11ee\x2d95cd\x2d87774f69a715.slice/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service 2024-03-24T11:50:00.364 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Non-zero exit code 1 from systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: Traceback (most recent call last): 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: return _run_code(code, main_globals, None, 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: exec(code, run_globals) 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:50:00.365 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1029, in deploy_daemon 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1165, in deploy_daemon_units 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: RuntimeError: Failed command: systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[64662]: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1165, in deploy_daemon_units 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: RuntimeError: Failed command: systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:50:00.366 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:00 smithi145 ceph-mon[67080]: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:50:00.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-24T11:50:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/295854291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:50:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-24T11:50:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: daemon iscsi.foo.smithi145.hnqhcg on smithi145 is in unknown state 2024-03-24T11:50:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi145.hnqhcg on host smithi145 failed. 2024-03-24T11:50:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Upgrade daemon: iscsi.foo.smithi145.hnqhcg: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi145.hnqhcg ... 2024-03-24T11:50:00.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Creating ceph-iscsi config... 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/iscsi-gateway.cfg 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Write file: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/iscsi.foo.smithi145.hnqhcg/tcmu-runner-entrypoint.sh 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2da2bb64be\x2de9d1\x2d11ee\x2d95cd\x2d87774f69a715.slice/ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Non-zero exit code 1 from systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: systemctl: stderr Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: systemctl: stderr See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: Traceback (most recent call last): 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: return _run_code(code, main_globals, None, 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: exec(code, run_globals) 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5520, in 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 5508, in main 2024-03-24T11:50:00.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3038, in command_deploy_from 2024-03-24T11:50:00.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3073, in _common_deploy 2024-03-24T11:50:00.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 3093, in _deploy_daemon_container 2024-03-24T11:50:00.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1029, in deploy_daemon 2024-03-24T11:50:00.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/__main__.py", line 1165, in deploy_daemon_units 2024-03-24T11:50:00.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: File "/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/cephadm.1da7fdc466b4c6904edf167cf65a395d23162ea50272e1f853080bb41ee7cd4a/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-24T11:50:00.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: RuntimeError: Failed command: systemctl start ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg: Job for ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service failed because the control process exited with error code. 2024-03-24T11:50:00.477 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:00 smithi149 ceph-mon[58107]: See "systemctl status ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" and "journalctl -xeu ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@iscsi.foo.smithi145.hnqhcg.service" for details. 2024-03-24T11:50:01.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:01 smithi145 ceph-mon[64662]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:01.362 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:01 smithi145 ceph-mon[67080]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:01.376 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:01 smithi149 ceph-mon[58107]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:02.426 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)" 2024-03-24T11:50:03.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-24T11:50:03.325 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:03 smithi145 ceph-mon[64662]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:03.325 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:03 smithi145 ceph-mon[64662]: from='client.? 172.21.15.145:0/2124131317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:03.325 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:03 smithi145 ceph-mon[67080]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:03.325 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:03 smithi145 ceph-mon[67080]: from='client.? 172.21.15.145:0/2124131317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:03.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:03 smithi149 ceph-mon[58107]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:03.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:03 smithi149 ceph-mon[58107]: from='client.? 172.21.15.145:0/2124131317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:50:05.024 INFO:teuthology.orchestra.run.smithi145.stdout:osd 8 106s ago - 2024-03-24T11:50:05.338 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:05 smithi145 ceph-mon[64662]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:05.338 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:05 smithi145 ceph-mon[64662]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:05.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:05 smithi145 ceph-mon[67080]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:05.338 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:05 smithi145 ceph-mon[67080]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:05.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:05 smithi149 ceph-mon[58107]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:05.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:05 smithi149 ceph-mon[58107]: from='mgr.34107 172.21.15.145:0/2659270978' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:05.611 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:05 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:50:05] "GET /metrics HTTP/1.1" 200 38709 "" "Prometheus/2.33.4" 2024-03-24T11:50:05.721 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:50:05.730 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-24T11:50:05.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-24T11:50:06.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:06 smithi145 ceph-mon[64662]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:06.342 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:06 smithi145 ceph-mon[67080]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:06.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:06 smithi149 ceph-mon[58107]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:07.340 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:07 smithi145 ceph-mon[67080]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:07.340 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:07 smithi145 ceph-mon[67080]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:07.340 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:07 smithi145 ceph-mon[64662]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:07.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:07 smithi145 ceph-mon[64662]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:07 smithi149 ceph-mon[58107]: from='client.44674 -' entity='client.iscsi.foo.smithi145.hnqhcg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:50:07.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:07 smithi149 ceph-mon[58107]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:08.140 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-24T11:50:08.140 INFO:teuthology.orchestra.run.smithi145.stdout: "image": "quay.io/ceph/ceph", 2024-03-24T11:50:08.140 INFO:teuthology.orchestra.run.smithi145.stdout: "registry": "quay.io", 2024-03-24T11:50:08.140 INFO:teuthology.orchestra.run.smithi145.stdout: "bare_image": "ceph/ceph", 2024-03-24T11:50:08.140 INFO:teuthology.orchestra.run.smithi145.stdout: "versions": [] 2024-03-24T11:50:08.140 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-24T11:50:08.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:08 smithi149 ceph-mon[58107]: from='client.44791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:08.494 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:08 smithi145 ceph-mon[67080]: from='client.44791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:08.494 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:08 smithi145 ceph-mon[64662]: from='client.44791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:08.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-24T11:50:09.322 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:09 smithi145 ceph-mon[64662]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:09.322 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:09 smithi145 ceph-mon[67080]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:09.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:09 smithi149 ceph-mon[58107]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:11.377 INFO:teuthology.orchestra.run.smithi145.stdout: "16.2.0", 2024-03-24T11:50:11.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:11 smithi149 ceph-mon[58107]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:11.611 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:11 smithi145 ceph-mon[64662]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:11.611 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:11 smithi145 ceph-mon[67080]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:12.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-24T11:50:12.315 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:12 smithi145 ceph-mon[64662]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:12.315 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:12 smithi145 ceph-mon[67080]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:12.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:12 smithi149 ceph-mon[58107]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:13.265 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:13 smithi145 ceph-mon[67080]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:13.265 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:13 smithi145 ceph-mon[64662]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:13.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:13 smithi149 ceph-mon[58107]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:50:14.588 INFO:teuthology.orchestra.run.smithi145.stdout: "v16.2.2", 2024-03-24T11:50:14.589 INFO:teuthology.orchestra.run.smithi145.stdout: "v16.2.2-20210505", 2024-03-24T11:50:15.079 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T11:50:15.090 INFO:tasks.cephadm:Teardown begin 2024-03-24T11:50:15.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:15.119 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:15.155 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T11:50:15.155 DEBUG:teuthology.orchestra.run.smithi145:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:50:15.174 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:50:15.212 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T11:50:15.212 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T11:50:15.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:50:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:15 smithi149 ceph-mon[58107]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:15.475 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:15 smithi149 ceph-mon[58107]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:15.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-mon[64662]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:15.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-mon[64662]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:15.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:15 smithi145 systemd[1]: Stopping Ceph mon.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:15.527 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-mon[67080]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:15.527 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-mon[67080]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:50:15.801 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a[64658]: 2024-03-24T11:50:15.525+0000 7f287e7d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a[64658]: 2024-03-24T11:50:15.525+0000 7f287e7d2700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: ::ffff:172.21.15.149 - - [24/Mar/2024:11:50:15] "GET /metrics HTTP/1.1" 200 38709 "" "Prometheus/2.33.4" 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:15] ENGINE Bus STOPPING 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:15] ENGINE Bus STOPPED 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:15] ENGINE Bus STARTING 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: CherryPy Checker: 2024-03-24T11:50:15.802 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: The Application mounted at '' has an empty config. 2024-03-24T11:50:15.803 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:50:15.803 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:15] ENGINE Serving on http://:::9283 2024-03-24T11:50:15.803 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:15 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:15] ENGINE Bus STARTED 2024-03-24T11:50:16.100 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:15 smithi145 podman[91331]: 2024-03-24 11:50:15.801580431 +0000 UTC m=+0.404956979 container died 7ad6536c6cf9d5bb70bf6831973cbd38966d2840abfc03f66b0974f85a43f11a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T11:50:16.101 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:16 smithi145 podman[91331]: 2024-03-24 11:50:16.100819106 +0000 UTC m=+0.704195653 container cleanup 7ad6536c6cf9d5bb70bf6831973cbd38966d2840abfc03f66b0974f85a43f11a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:50:16.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:16 smithi145 bash[91331]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:50:16.361 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 24 11:50:16 smithi145 podman[91355]: 2024-03-24 11:50:16.335374806 +0000 UTC m=+0.530600747 container remove 7ad6536c6cf9d5bb70bf6831973cbd38966d2840abfc03f66b0974f85a43f11a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T11:50:16.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T11:50:16.496 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:16.496 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T11:50:16.497 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-24T11:50:16.497 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.c 2024-03-24T11:50:16.788 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:16 smithi145 systemd[1]: Stopping Ceph mon.c for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c[67076]: 2024-03-24T11:50:16.787+0000 7f17cd434700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c[67076]: 2024-03-24T11:50:16.787+0000 7f17cd434700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-24T11:50:17.111 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:16 smithi145 podman[91467]: 2024-03-24 11:50:16.819551107 +0000 UTC m=+0.141492835 container died 4c983fe930c037b40829d97f2ede0f067586e2ca8f49d2d1bf5633b083f07e6a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T11:50:17.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:16] ENGINE Bus STOPPING 2024-03-24T11:50:17.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:50:17.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:16] ENGINE Bus STOPPED 2024-03-24T11:50:17.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:16] ENGINE Bus STARTING 2024-03-24T11:50:17.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:16] ENGINE Serving on http://:::9283 2024-03-24T11:50:17.112 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:16 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: [24/Mar/2024:11:50:16] ENGINE Bus STARTED 2024-03-24T11:50:17.448 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:17 smithi145 podman[91467]: 2024-03-24 11:50:17.205679913 +0000 UTC m=+0.527621677 container cleanup 4c983fe930c037b40829d97f2ede0f067586e2ca8f49d2d1bf5633b083f07e6a (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T11:50:17.448 INFO:journalctl@ceph.mon.c.smithi145.stdout:Mar 24 11:50:17 smithi145 bash[91467]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-c 2024-03-24T11:50:17.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T11:50:17.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:17.614 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-24T11:50:17.614 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-24T11:50:17.614 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.b 2024-03-24T11:50:17.975 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:17 smithi149 systemd[1]: Stopping Ceph mon.b for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:18.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b[58102]: 2024-03-24T11:50:18.000+0000 7f76f2340700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:18.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b[58102]: 2024-03-24T11:50:18.000+0000 7f76f2340700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-24T11:50:18.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:18 smithi149 podman[76605]: 2024-03-24 11:50:18.1547762 +0000 UTC m=+0.344933106 container died 6dedc9190b9eb1bb4c4ba2318c6d420f6c55e861d5fa9fe14f9c7453cca3c553 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:50:18.724 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:18 smithi149 podman[76605]: 2024-03-24 11:50:18.618206217 +0000 UTC m=+0.808363192 container cleanup 6dedc9190b9eb1bb4c4ba2318c6d420f6c55e861d5fa9fe14f9c7453cca3c553 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T11:50:18.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 24 11:50:18 smithi149 bash[76605]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mon-b 2024-03-24T11:50:19.093 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T11:50:19.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:19.138 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-24T11:50:19.138 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T11:50:19.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:50:19.434 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:19 smithi145 systemd[1]: Stopping Ceph mgr.y for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:50:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:50:19.462Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=ceph-exporter\": read tcp 172.21.15.149:60088->172.21.15.145:8765: read: connection reset by peer" 2024-03-24T11:50:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:50:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:50:19.462Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=mgr-prometheus\": read tcp 172.21.15.149:60118->172.21.15.145:8765: read: connection reset by peer" 2024-03-24T11:50:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:50:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:50:19.462Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=node-exporter\": read tcp 172.21.15.149:60100->172.21.15.145:8765: read: connection reset by peer" 2024-03-24T11:50:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:50:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:50:19.462Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=alertmanager\": read tcp 172.21.15.149:60112->172.21.15.145:8765: read: connection reset by peer" 2024-03-24T11:50:19.744 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:19 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:50:19.433+0000 7f5039ae7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:19.744 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:19 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y[60235]: 2024-03-24T11:50:19.433+0000 7f5039ae7700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T11:50:19.744 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:19 smithi145 podman[91599]: 2024-03-24 11:50:19.465335613 +0000 UTC m=+0.130013069 container died 387fa4d011fea73b0e1e11d44a57022473cea2a2bd2cd6ee257e60b5cc0b372d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-db0330b) 2024-03-24T11:50:20.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:19 smithi145 podman[91599]: 2024-03-24 11:50:19.834795621 +0000 UTC m=+0.499473077 container cleanup 387fa4d011fea73b0e1e11d44a57022473cea2a2bd2cd6ee257e60b5cc0b372d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T11:50:20.111 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Mar 24 11:50:19 smithi145 bash[91599]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-y 2024-03-24T11:50:20.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T11:50:20.293 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:20.293 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T11:50:20.294 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T11:50:20.294 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x 2024-03-24T11:50:20.713 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:50:20 smithi149 systemd[1]: Stopping Ceph mgr.x for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:20.973 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:50:20 smithi149 podman[76724]: 2024-03-24 11:50:20.712942862 +0000 UTC m=+0.224813064 container died bba2809eb6bea83a904974bad2f3e70b231771b056648e0ba44fcbf23e3aea87 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-24T11:50:21.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:50:21 smithi149 podman[76724]: 2024-03-24 11:50:21.072854516 +0000 UTC m=+0.584724701 container cleanup bba2809eb6bea83a904974bad2f3e70b231771b056648e0ba44fcbf23e3aea87 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:50:21.225 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 24 11:50:21 smithi149 bash[76724]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-mgr-x 2024-03-24T11:50:21.549 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T11:50:21.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:21.586 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T11:50:21.586 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T11:50:21.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.0 2024-03-24T11:50:21.905 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:21 smithi145 systemd[1]: Stopping Ceph osd.0 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:22.361 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[70436]: 2024-03-24T11:50:21.905+0000 7f756461d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:22.361 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[70436]: 2024-03-24T11:50:21.905+0000 7f756461d700 -1 osd.0 153 *** Got signal Terminated *** 2024-03-24T11:50:22.361 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:21 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0[70436]: 2024-03-24T11:50:21.905+0000 7f756461d700 -1 osd.0 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:50:23.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:23.548Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:23.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:23.549Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:27.514 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:27 smithi145 podman[91721]: 2024-03-24 11:50:27.241365749 +0000 UTC m=+5.458219026 container died b5960e0232afd0815c23f5d2829dc8acc52dc0156c58f6dae573ec809a9c1d1e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T11:50:27.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:27 smithi145 podman[91721]: 2024-03-24 11:50:27.622123834 +0000 UTC m=+5.838977113 container cleanup b5960e0232afd0815c23f5d2829dc8acc52dc0156c58f6dae573ec809a9c1d1e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T11:50:27.861 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:27 smithi145 bash[91721]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:50:28.194 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:27 smithi145 podman[91738]: 2024-03-24 11:50:27.873602902 +0000 UTC m=+0.627402122 container remove b5960e0232afd0815c23f5d2829dc8acc52dc0156c58f6dae573ec809a9c1d1e (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:50:28.485 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:28 smithi145 podman[91833]: 2024-03-24 11:50:28.09860562 +0000 UTC m=+0.018119110 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:50:28.485 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:28 smithi145 podman[91833]: 2024-03-24 11:50:28.199312113 +0000 UTC m=+0.118825607 container create 0861849acc0dca8461a3e2a2d38791805022a4a600f0c19dba583c0767960fe8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:50:28.485 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:28 smithi145 podman[91833]: 2024-03-24 11:50:28.48524158 +0000 UTC m=+0.404755065 container init 0861849acc0dca8461a3e2a2d38791805022a4a600f0c19dba583c0767960fe8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-03-24T11:50:28.779 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:28 smithi145 podman[91833]: 2024-03-24 11:50:28.48879993 +0000 UTC m=+0.408313422 container start 0861849acc0dca8461a3e2a2d38791805022a4a600f0c19dba583c0767960fe8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T11:50:28.779 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:28 smithi145 podman[91833]: 2024-03-24 11:50:28.593725811 +0000 UTC m=+0.513239316 container attach 0861849acc0dca8461a3e2a2d38791805022a4a600f0c19dba583c0767960fe8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:50:29.055 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:28 smithi145 podman[91833]: 2024-03-24 11:50:28.778824398 +0000 UTC m=+0.698337885 container died 0861849acc0dca8461a3e2a2d38791805022a4a600f0c19dba583c0767960fe8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T11:50:29.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T11:50:29.318 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:29 smithi145 podman[91860]: 2024-03-24 11:50:29.272286301 +0000 UTC m=+0.488219137 container remove 0861849acc0dca8461a3e2a2d38791805022a4a600f0c19dba583c0767960fe8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD) 2024-03-24T11:50:29.318 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:29 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T11:50:29.318 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:29 smithi145 systemd[1]: Stopped Ceph osd.0 for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:50:29.318 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 24 11:50:29 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.0.service: Consumed 6.861s CPU time. 2024-03-24T11:50:29.334 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:29.334 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T11:50:29.334 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T11:50:29.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.1 2024-03-24T11:50:29.580 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:29 smithi145 systemd[1]: Stopping Ceph osd.1 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:29.858 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:29 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[73064]: 2024-03-24T11:50:29.725+0000 7f2e5f2f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:29.858 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:29 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[73064]: 2024-03-24T11:50:29.725+0000 7f2e5f2f8700 -1 osd.1 153 *** Got signal Terminated *** 2024-03-24T11:50:29.858 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:29 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1[73064]: 2024-03-24T11:50:29.725+0000 7f2e5f2f8700 -1 osd.1 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:50:33.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:50:33.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:33.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:33.548Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:33.862 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:33 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:33.548Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:34.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:34.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:34.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:34 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:34.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:35.367 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:35 smithi145 podman[91941]: 2024-03-24 11:50:35.111704854 +0000 UTC m=+5.515617206 container died 848b03020bb19af4f4fdcaad341455ca1ce983c598d3a7db61a1643f6db49802 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T11:50:35.725 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:35 smithi145 podman[91941]: 2024-03-24 11:50:35.466544634 +0000 UTC m=+5.870456985 container cleanup 848b03020bb19af4f4fdcaad341455ca1ce983c598d3a7db61a1643f6db49802 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T11:50:35.725 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:35 smithi145 bash[91941]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:50:36.089 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:35 smithi145 podman[91970]: 2024-03-24 11:50:35.726037393 +0000 UTC m=+0.612230485 container remove 848b03020bb19af4f4fdcaad341455ca1ce983c598d3a7db61a1643f6db49802 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T11:50:36.361 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:36 smithi145 podman[92061]: 2024-03-24 11:50:35.993751134 +0000 UTC m=+0.018229394 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:50:36.361 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:36 smithi145 podman[92061]: 2024-03-24 11:50:36.110519367 +0000 UTC m=+0.134997668 container create 8c2ef5067bb8f3a29c8ee2fb8044cb6122cd2911b152e6a8f4747858ee2f0cb8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:50:36.361 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:36 smithi145 podman[92061]: 2024-03-24 11:50:36.329712471 +0000 UTC m=+0.354190726 container init 8c2ef5067bb8f3a29c8ee2fb8044cb6122cd2911b152e6a8f4747858ee2f0cb8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-24T11:50:36.361 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:36 smithi145 podman[92061]: 2024-03-24 11:50:36.33293693 +0000 UTC m=+0.357415188 container start 8c2ef5067bb8f3a29c8ee2fb8044cb6122cd2911b152e6a8f4747858ee2f0cb8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T11:50:36.861 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:36 smithi145 podman[92061]: 2024-03-24 11:50:36.413074857 +0000 UTC m=+0.437553126 container attach 8c2ef5067bb8f3a29c8ee2fb8044cb6122cd2911b152e6a8f4747858ee2f0cb8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:50:36.861 INFO:journalctl@ceph.osd.1.smithi145.stdout:Mar 24 11:50:36 smithi145 podman[92061]: 2024-03-24 11:50:36.620521705 +0000 UTC m=+0.644999952 container died 8c2ef5067bb8f3a29c8ee2fb8044cb6122cd2911b152e6a8f4747858ee2f0cb8 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T11:50:37.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T11:50:37.252 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:37.252 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T11:50:37.252 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T11:50:37.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.2 2024-03-24T11:50:37.578 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:37 smithi145 systemd[1]: Stopping Ceph osd.2 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:37.861 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:37 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[75546]: 2024-03-24T11:50:37.577+0000 7fbfa9c7a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:37.861 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:37 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[75546]: 2024-03-24T11:50:37.577+0000 7fbfa9c7a700 -1 osd.2 153 *** Got signal Terminated *** 2024-03-24T11:50:37.861 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:37 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2[75546]: 2024-03-24T11:50:37.577+0000 7fbfa9c7a700 -1 osd.2 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:50:43.327 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:43 smithi145 podman[92167]: 2024-03-24 11:50:43.051774238 +0000 UTC m=+5.575868824 container died a91a34aa846516a51e733d3bcaba70625cf16d6fb91c3ea8895bd60a4fc415d4 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:50:43.327 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:43 smithi145 podman[92167]: 2024-03-24 11:50:43.327380054 +0000 UTC m=+5.851474650 container cleanup a91a34aa846516a51e733d3bcaba70625cf16d6fb91c3ea8895bd60a4fc415d4 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:50:43.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:43 smithi145 bash[92167]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:50:43.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:43 smithi145 podman[92183]: 2024-03-24 11:50:43.570246552 +0000 UTC m=+0.515641950 container remove a91a34aa846516a51e733d3bcaba70625cf16d6fb91c3ea8895bd60a4fc415d4 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-24T11:50:43.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:50:43.548Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:43.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:43.548Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:43.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:43 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:43.549Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:44.181 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:43 smithi145 podman[92279]: 2024-03-24 11:50:43.800806435 +0000 UTC m=+0.017867831 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:50:44.182 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:43 smithi145 podman[92279]: 2024-03-24 11:50:43.904759264 +0000 UTC m=+0.121820721 container create d6deb0debdd6453992670f62186f34e57c8f4b19e41f8aa2ff5c9206336c82c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-24T11:50:44.182 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:44 smithi145 podman[92279]: 2024-03-24 11:50:44.124134368 +0000 UTC m=+0.341195765 container init d6deb0debdd6453992670f62186f34e57c8f4b19e41f8aa2ff5c9206336c82c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b) 2024-03-24T11:50:44.182 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:44 smithi145 podman[92279]: 2024-03-24 11:50:44.12753515 +0000 UTC m=+0.344596542 container start d6deb0debdd6453992670f62186f34e57c8f4b19e41f8aa2ff5c9206336c82c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T11:50:44.182 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:44 smithi145 podman[92279]: 2024-03-24 11:50:44.182068169 +0000 UTC m=+0.399129569 container attach d6deb0debdd6453992670f62186f34e57c8f4b19e41f8aa2ff5c9206336c82c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-24T11:50:44.611 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:44 smithi145 podman[92279]: 2024-03-24 11:50:44.41751319 +0000 UTC m=+0.634574581 container died d6deb0debdd6453992670f62186f34e57c8f4b19e41f8aa2ff5c9206336c82c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-db0330b) 2024-03-24T11:50:44.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:50:44.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:44.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:44.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:44.612 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:44 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:44.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:44.957 INFO:journalctl@ceph.osd.2.smithi145.stdout:Mar 24 11:50:44 smithi145 podman[92304]: 2024-03-24 11:50:44.943930231 +0000 UTC m=+0.520764654 container remove d6deb0debdd6453992670f62186f34e57c8f4b19e41f8aa2ff5c9206336c82c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:50:44.962 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T11:50:45.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:45.004 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T11:50:45.005 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T11:50:45.005 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.3 2024-03-24T11:50:45.330 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:45 smithi145 systemd[1]: Stopping Ceph osd.3 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:45.611 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[78013]: 2024-03-24T11:50:45.329+0000 7fb6c35fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:45.611 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[78013]: 2024-03-24T11:50:45.329+0000 7fb6c35fd700 -1 osd.3 153 *** Got signal Terminated *** 2024-03-24T11:50:45.611 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:45 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3[78013]: 2024-03-24T11:50:45.329+0000 7fb6c35fd700 -1 osd.3 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:50:48.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:48.131+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:48.880 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:48 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:48.616+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:49.224 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:49.110+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:49.611 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:49.226+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:49.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:49.621+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:49.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:49 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:49.753+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:50.474 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:50.206+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:50.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:50.120+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:50.974 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:50.664+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:50.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:50 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:50.788+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:51.111 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:50 smithi145 podman[92384]: 2024-03-24 11:50:50.8213277 +0000 UTC m=+5.589751268 container died ffbb9bef1b39b18e1035b8f8d2360692f0b66eb9b7ceea3a93f5f357c0703ab9 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:50:51.380 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 podman[92384]: 2024-03-24 11:50:51.121226656 +0000 UTC m=+5.889650232 container cleanup ffbb9bef1b39b18e1035b8f8d2360692f0b66eb9b7ceea3a93f5f357c0703ab9 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T11:50:51.380 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 bash[92384]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3 2024-03-24T11:50:51.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:51.120+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:51.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:51.192+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:51.697 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 podman[92402]: 2024-03-24 11:50:51.380600272 +0000 UTC m=+0.554131050 container remove ffbb9bef1b39b18e1035b8f8d2360692f0b66eb9b7ceea3a93f5f357c0703ab9 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b) 2024-03-24T11:50:51.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:51.699+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:51.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:51 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:51.786+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:51.975 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 podman[92495]: 2024-03-24 11:50:51.69809708 +0000 UTC m=+0.114128267 container create e7422506a2543c1d655701b9332882446e321930a1f4eb2ed746b48efa314e3c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2) 2024-03-24T11:50:51.975 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 podman[92495]: 2024-03-24 11:50:51.602704477 +0000 UTC m=+0.018735620 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:50:51.976 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 podman[92495]: 2024-03-24 11:50:51.900449389 +0000 UTC m=+0.316480528 container init e7422506a2543c1d655701b9332882446e321930a1f4eb2ed746b48efa314e3c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:50:51.976 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 podman[92495]: 2024-03-24 11:50:51.903615469 +0000 UTC m=+0.319646608 container start e7422506a2543c1d655701b9332882446e321930a1f4eb2ed746b48efa314e3c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:50:51.976 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:51 smithi145 podman[92495]: 2024-03-24 11:50:51.97543066 +0000 UTC m=+0.391461813 container attach e7422506a2543c1d655701b9332882446e321930a1f4eb2ed746b48efa314e3c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T11:50:52.361 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:52 smithi145 podman[92495]: 2024-03-24 11:50:52.190449692 +0000 UTC m=+0.606480831 container died e7422506a2543c1d655701b9332882446e321930a1f4eb2ed746b48efa314e3c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:50:52.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:52.128+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:52.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:52.192+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:52.642 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 24 11:50:52 smithi145 podman[92521]: 2024-03-24 11:50:52.620602113 +0000 UTC m=+0.424167391 container remove e7422506a2543c1d655701b9332882446e321930a1f4eb2ed746b48efa314e3c (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-24T11:50:52.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T11:50:52.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:52.688 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T11:50:52.688 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T11:50:52.688 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.4 2024-03-24T11:50:52.975 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:52 smithi149 systemd[1]: Stopping Ceph osd.4 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:50:52.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:52.648+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:52.976 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:52.814+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:53.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:53 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:53.215+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:53.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:52.997+0000 7f6af71c6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:53.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:52.997+0000 7f6af71c6700 -1 osd.4 153 *** Got signal Terminated *** 2024-03-24T11:50:53.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:52 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:52.997+0000 7f6af71c6700 -1 osd.4 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:50:53.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:53 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:53.140+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:53.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:50:53.548Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:53.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:53.549Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:53.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:53 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:53.549Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:53.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:53 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:53.781+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:53.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:53 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:53.628+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:54.474 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:54 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:54.178+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:54.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:54 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:54.121+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:54.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:50:54.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:54.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:54.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:50:54.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:50:54 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:50:54.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:50:54.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:54 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:54.810+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:54.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:54 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:54.652+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:55.474 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:55 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:55.216+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:55.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:55 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:55.168+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:55.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:55 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:55.168+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:32.204257+0000 front 2024-03-24T11:50:32.204357+0000 (oldest deadline 2024-03-24T11:50:55.104284+0000) 2024-03-24T11:50:55.974 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:55 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:55.811+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:55.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:55 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:55.638+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:55.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:55 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:55.638+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:50:56.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:56 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:56.255+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:56.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:56 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:56.182+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:56.476 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:56 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:56.182+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:32.204257+0000 front 2024-03-24T11:50:32.204357+0000 (oldest deadline 2024-03-24T11:50:55.104284+0000) 2024-03-24T11:50:56.974 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:56 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:56.814+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:56.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:56 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:56.593+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:56.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:56 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:56.593+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:50:57.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:57.303+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:57.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:57.196+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:22.803541+0000 front 2024-03-24T11:50:22.803423+0000 (oldest deadline 2024-03-24T11:50:48.103407+0000) 2024-03-24T11:50:57.475 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4[61389]: 2024-03-24T11:50:57.196+0000 7f6aef7b2700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:32.204257+0000 front 2024-03-24T11:50:32.204357+0000 (oldest deadline 2024-03-24T11:50:55.104284+0000) 2024-03-24T11:50:57.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:57.608+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:57.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:57.608+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:50:57.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:57.766+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:57.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:57 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:57.766+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:50:58.600 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:58 smithi149 podman[76861]: 2024-03-24 11:50:58.327595091 +0000 UTC m=+5.449937274 container died 8a8a001df0e15bda346dca747effce02cb67fd2eb15eb68d0d63436bbbeb5757 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-24T11:50:58.600 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:58.314+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:58.600 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:58.314+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:50:58.967 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:58.653+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:58.967 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:58.653+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:50:58.967 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:58.777+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:58.967 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:58 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:58.777+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:50:58.967 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:58 smithi149 podman[76861]: 2024-03-24 11:50:58.765646104 +0000 UTC m=+5.887988286 container cleanup 8a8a001df0e15bda346dca747effce02cb67fd2eb15eb68d0d63436bbbeb5757 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T11:50:58.968 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:58 smithi149 bash[76861]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4 2024-03-24T11:50:59.225 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:58 smithi149 podman[76878]: 2024-03-24 11:50:58.96728085 +0000 UTC m=+0.637166683 container remove 8a8a001df0e15bda346dca747effce02cb67fd2eb15eb68d0d63436bbbeb5757 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-24T11:50:59.612 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:59 smithi149 podman[76968]: 2024-03-24 11:50:59.281494426 +0000 UTC m=+0.018758545 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:50:59.612 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:59 smithi149 podman[76968]: 2024-03-24 11:50:59.401458852 +0000 UTC m=+0.138722961 container create c16043dd44cea5c7eed526ed5aa9acdc1e07b971258abac8430f8776034550dc (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux ) 2024-03-24T11:50:59.612 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:59 smithi149 podman[76968]: 2024-03-24 11:50:59.612146584 +0000 UTC m=+0.349410707 container init c16043dd44cea5c7eed526ed5aa9acdc1e07b971258abac8430f8776034550dc (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T11:50:59.612 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:59 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:59.331+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:50:59.612 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:50:59 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:50:59.331+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:50:59.936 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:59 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:59.748+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:50:59.936 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:50:59 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:50:59.748+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:50:59.937 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:59 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:59.680+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:50:59.937 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:50:59 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:50:59.680+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:50:59.937 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:59 smithi149 podman[76968]: 2024-03-24 11:50:59.615287413 +0000 UTC m=+0.352551523 container start c16043dd44cea5c7eed526ed5aa9acdc1e07b971258abac8430f8776034550dc (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T11:50:59.937 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:59 smithi149 podman[76968]: 2024-03-24 11:50:59.72915509 +0000 UTC m=+0.466419207 container attach c16043dd44cea5c7eed526ed5aa9acdc1e07b971258abac8430f8776034550dc (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T11:51:00.224 INFO:journalctl@ceph.osd.4.smithi149.stdout:Mar 24 11:50:59 smithi149 podman[76968]: 2024-03-24 11:50:59.93729356 +0000 UTC m=+0.674557680 container died c16043dd44cea5c7eed526ed5aa9acdc1e07b971258abac8430f8776034550dc (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-24T11:51:00.624 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:00 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:00.363+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:51:00.625 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:00 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:00.363+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:51:00.644 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T11:51:00.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:00.690 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T11:51:00.690 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T11:51:00.690 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.5 2024-03-24T11:51:00.898 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:00 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:00.728+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:00.898 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:00 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:00.728+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:00.899 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:00 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:00.733+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:00.899 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:00 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:00.733+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:00.899 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:00 smithi149 systemd[1]: Stopping Ceph osd.5 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:51:01.224 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:01.069+0000 7fb2e83db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:51:01.225 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:01.069+0000 7fb2e83db700 -1 osd.5 153 *** Got signal Terminated *** 2024-03-24T11:51:01.225 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:01.069+0000 7fb2e83db700 -1 osd.5 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:51:01.689 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:01.377+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:51:01.689 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:01.377+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:51:01.974 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:01.688+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:01.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:01.688+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:01.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:01.760+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:01.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:01 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:01.760+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:02.690 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:02.335+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:51:02.691 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:02.335+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:51:02.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:02.690+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:02.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:02.690+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:02.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:02.720+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:02.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:02 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:02.720+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:03.695 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:03.303+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:51:03.696 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:03.303+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:51:03.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:51:03.549Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:03.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:03.550Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:03.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:03 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:03.550Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:03.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:03.694+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:03.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:03.694+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:03.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:03.722+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:03.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:03 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:03.722+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:04.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:51:04.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:04.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:04.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:04.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:04 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:04.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:04.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:04.297+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:51:04.725 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:04.297+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:51:05.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:04.726+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:05.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:04.726+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:05.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:04.741+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:05.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:04 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:04.741+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:05.709 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:05.284+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:25.073516+0000 front 2024-03-24T11:50:25.073386+0000 (oldest deadline 2024-03-24T11:50:49.173449+0000) 2024-03-24T11:51:05.710 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5[63734]: 2024-03-24T11:51:05.284+0000 7fb2e49cf700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:29.173910+0000 front 2024-03-24T11:50:29.174048+0000 (oldest deadline 2024-03-24T11:50:57.974309+0000) 2024-03-24T11:51:05.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:05.709+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:05.975 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:05.709+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:05.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:05.764+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:05.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:05.764+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:06.887 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:06 smithi149 podman[77074]: 2024-03-24 11:51:06.584769084 +0000 UTC m=+5.644975556 container died ef3114caac79b70d5c0b85b32f24aa5d125bff0a1fa8cce0c53d6812cc91f270 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T11:51:06.887 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:06.727+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:06.887 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:06.727+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:06.888 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:06.727+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:06.888 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:06.769+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:06.888 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:06 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:06.769+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:07.225 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:07 smithi149 podman[77074]: 2024-03-24 11:51:07.040479449 +0000 UTC m=+6.100685915 container cleanup ef3114caac79b70d5c0b85b32f24aa5d125bff0a1fa8cce0c53d6812cc91f270 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T11:51:07.225 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:07 smithi149 bash[77074]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5 2024-03-24T11:51:07.652 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:07 smithi149 podman[77102]: 2024-03-24 11:51:07.333521727 +0000 UTC m=+0.744978735 container remove ef3114caac79b70d5c0b85b32f24aa5d125bff0a1fa8cce0c53d6812cc91f270 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:51:07.912 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:07 smithi149 podman[77197]: 2024-03-24 11:51:07.555136519 +0000 UTC m=+0.017696256 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:51:07.912 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:07 smithi149 podman[77197]: 2024-03-24 11:51:07.67598697 +0000 UTC m=+0.138546703 container create f561b8c01855e79f2c5bfb026ca601a132b17749fe929881bf52ca60a87c68c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T11:51:07.912 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:07.770+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:07.913 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:07.770+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:07.913 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:07.770+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:07.913 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:07.720+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:07.913 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:07.720+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:07.913 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:07 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:07.720+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:08.207 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:07 smithi149 podman[77197]: 2024-03-24 11:51:07.912541643 +0000 UTC m=+0.375101372 container init f561b8c01855e79f2c5bfb026ca601a132b17749fe929881bf52ca60a87c68c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-03-24T11:51:08.207 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:07 smithi149 podman[77197]: 2024-03-24 11:51:07.91619964 +0000 UTC m=+0.378759375 container start f561b8c01855e79f2c5bfb026ca601a132b17749fe929881bf52ca60a87c68c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:51:08.207 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:08 smithi149 podman[77197]: 2024-03-24 11:51:08.003586698 +0000 UTC m=+0.466146433 container attach f561b8c01855e79f2c5bfb026ca601a132b17749fe929881bf52ca60a87c68c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T11:51:08.475 INFO:journalctl@ceph.osd.5.smithi149.stdout:Mar 24 11:51:08 smithi149 podman[77197]: 2024-03-24 11:51:08.207418001 +0000 UTC m=+0.669977736 container died f561b8c01855e79f2c5bfb026ca601a132b17749fe929881bf52ca60a87c68c2 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T11:51:08.796 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:08.747+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:08.796 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:08.747+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:08.796 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:08.747+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:08.954 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T11:51:08.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:08.999 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T11:51:09.000 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T11:51:09.000 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.6 2024-03-24T11:51:09.224 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:08.794+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:09.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:08.794+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:09.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:08 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:08.794+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:09.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:09 smithi149 systemd[1]: Stopping Ceph osd.6 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:51:09.725 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:09.317+0000 7efd3e802700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:51:09.725 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:09.317+0000 7efd3e802700 -1 osd.6 153 *** Got signal Terminated *** 2024-03-24T11:51:09.725 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:09.317+0000 7efd3e802700 -1 osd.6 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:51:10.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:09.798+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:10.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:09.798+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:10.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:09.798+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:10.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:09.756+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:10.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:09.756+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:10.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:09 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:09.756+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:11.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:10.800+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:11.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:10.800+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:11.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:10.800+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:11.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:10.796+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:11.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:10.796+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:11.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:10 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:10.796+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:12.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:11.779+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:12.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:11.779+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:12.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:11.779+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:12.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:11.753+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:12.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:11.753+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:12.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:11.753+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:12.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:11 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:11.753+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:13.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:13.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:13.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:13.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:49.814508+0000 front 2024-03-24T11:50:49.814499+0000 (oldest deadline 2024-03-24T11:51:12.714387+0000) 2024-03-24T11:51:13.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:12.759+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:13.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:12.759+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:13.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:12.759+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:13.226 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:12 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:12.759+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:13.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:13 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:51:13.549Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:13.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:13 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:13.550Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:13.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:13 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:13.551Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:49.814508+0000 front 2024-03-24T11:50:49.814499+0000 (oldest deadline 2024-03-24T11:51:12.714387+0000) 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:13.725+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:13.725+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:13.725+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:14.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:13 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:13.725+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:14.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:51:14.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:14.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:14.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:14.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:14 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:14.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:14.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:14.691+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:14.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:14.691+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:14.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:14.691+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:14.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:14.691+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:14.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: *** Caught signal (Segmentation fault) ** 2024-03-24T11:51:14.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: in thread 7efd29dd4700 thread_name:safe_timer 2024-03-24T11:51:14.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:51:14.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efd45c58d20] 2024-03-24T11:51:14.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x558a1817588a] 2024-03-24T11:51:14.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x558a18203668] 2024-03-24T11:51:14.976 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x558a17dea263] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x558a18074b81] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x558a17d65564] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x558a17d675ba] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x558a17d69523] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x558a17ccb0da] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x558a1832495f] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 11: (CommonSafeTimerThread::entry()+0x11) [0x558a18325c81] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efd45c4e1ca] 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 13: clone() 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:14.670+0000 7efd29dd4700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T11:51:14.977 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: in thread 7efd29dd4700 thread_name:safe_timer 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efd45c58d20] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x558a1817588a] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x558a18203668] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x558a17dea263] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x558a18074b81] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x558a17d65564] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x558a17d675ba] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x558a17d69523] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x558a17ccb0da] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x558a1832495f] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 11: (CommonSafeTimerThread::entry()+0x11) [0x558a18325c81] 2024-03-24T11:51:14.978 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efd45c4e1ca] 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 13: clone() 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -5362> 2024-03-24T11:50:48.616+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -4611> 2024-03-24T11:50:49.621+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -4453> 2024-03-24T11:50:50.664+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3810> 2024-03-24T11:50:51.699+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3752> 2024-03-24T11:50:52.648+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3645> 2024-03-24T11:50:53.628+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3558> 2024-03-24T11:50:54.652+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3484> 2024-03-24T11:50:55.638+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.979 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3483> 2024-03-24T11:50:55.638+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3347> 2024-03-24T11:50:56.593+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -3346> 2024-03-24T11:50:56.593+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2774> 2024-03-24T11:50:57.608+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2773> 2024-03-24T11:50:57.608+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2668> 2024-03-24T11:50:58.653+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2667> 2024-03-24T11:50:58.653+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2546> 2024-03-24T11:50:59.680+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2545> 2024-03-24T11:50:59.680+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2078> 2024-03-24T11:51:00.728+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -2077> 2024-03-24T11:51:00.728+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1964> 2024-03-24T11:51:01.688+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1963> 2024-03-24T11:51:01.688+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.980 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1922> 2024-03-24T11:51:02.690+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1921> 2024-03-24T11:51:02.690+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1819> 2024-03-24T11:51:03.722+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1818> 2024-03-24T11:51:03.722+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1415> 2024-03-24T11:51:04.741+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1414> 2024-03-24T11:51:04.741+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1314> 2024-03-24T11:51:05.709+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1313> 2024-03-24T11:51:05.709+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1224> 2024-03-24T11:51:06.727+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1223> 2024-03-24T11:51:06.727+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -1222> 2024-03-24T11:51:06.727+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.981 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -800> 2024-03-24T11:51:07.770+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -799> 2024-03-24T11:51:07.770+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -798> 2024-03-24T11:51:07.770+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -728> 2024-03-24T11:51:08.794+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -727> 2024-03-24T11:51:08.794+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -726> 2024-03-24T11:51:08.794+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -672> 2024-03-24T11:51:09.317+0000 7efd3e802700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -671> 2024-03-24T11:51:09.317+0000 7efd3e802700 -1 osd.6 153 *** Got signal Terminated *** 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -669> 2024-03-24T11:51:09.317+0000 7efd3e802700 -1 osd.6 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:51:14.982 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -609> 2024-03-24T11:51:09.798+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.983 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -608> 2024-03-24T11:51:09.798+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.983 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -607> 2024-03-24T11:51:09.798+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.983 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -564> 2024-03-24T11:51:10.800+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.983 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -563> 2024-03-24T11:51:10.800+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.983 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -562> 2024-03-24T11:51:10.800+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.983 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -510> 2024-03-24T11:51:11.779+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.983 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -509> 2024-03-24T11:51:11.779+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -508> 2024-03-24T11:51:11.779+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -355> 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -354> 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -353> 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -352> 2024-03-24T11:51:12.740+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:49.814508+0000 front 2024-03-24T11:50:49.814499+0000 (oldest deadline 2024-03-24T11:51:12.714387+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -196> 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.811001+0000 front 2024-03-24T11:50:24.811173+0000 (oldest deadline 2024-03-24T11:50:48.310970+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -195> 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.011972+0000 front 2024-03-24T11:50:33.011925+0000 (oldest deadline 2024-03-24T11:50:55.311888+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -194> 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:41.713487+0000 front 2024-03-24T11:50:41.713597+0000 (oldest deadline 2024-03-24T11:51:05.813434+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: -192> 2024-03-24T11:51:13.737+0000 7efd3adf6700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:49.814508+0000 front 2024-03-24T11:50:49.814499+0000 (oldest deadline 2024-03-24T11:51:12.714387+0000) 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 0> 2024-03-24T11:51:14.670+0000 7efd29dd4700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T11:51:14.984 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: in thread 7efd29dd4700 thread_name:safe_timer 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efd45c58d20] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x558a1817588a] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x558a18203668] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x558a17dea263] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x558a18074b81] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x558a17d65564] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x558a17d675ba] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x558a17d69523] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x558a17ccb0da] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x558a1832495f] 2024-03-24T11:51:14.985 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 11: (CommonSafeTimerThread::entry()+0x11) [0x558a18325c81] 2024-03-24T11:51:14.986 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efd45c4e1ca] 2024-03-24T11:51:14.986 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 13: clone() 2024-03-24T11:51:14.986 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T11:51:14.986 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6[66055]: 2024-03-24T11:51:14.986 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:14 smithi149 podman[77302]: 2024-03-24 11:51:14.737598741 +0000 UTC m=+5.554768038 container died 1223bf8d794a7da94f0553713f727797e16350c9c4372807ef3e8fd1515efc78 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T11:51:15.475 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:15 smithi149 podman[77302]: 2024-03-24 11:51:15.235880613 +0000 UTC m=+6.053049945 container cleanup 1223bf8d794a7da94f0553713f727797e16350c9c4372807ef3e8fd1515efc78 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T11:51:15.475 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:15 smithi149 bash[77302]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6 2024-03-24T11:51:15.856 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:15 smithi149 podman[77319]: 2024-03-24 11:51:15.512162866 +0000 UTC m=+0.776255407 container remove 1223bf8d794a7da94f0553713f727797e16350c9c4372807ef3e8fd1515efc78 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T11:51:15.857 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:15 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.6.service: Main process exited, code=exited, status=139/n/a 2024-03-24T11:51:15.857 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:15 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:15.663+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:15.857 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:15 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:15.663+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:15.857 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:15 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:15.663+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:15.857 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:15 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:15.663+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:16.165 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:15 smithi149 podman[77413]: 2024-03-24 11:51:15.761136873 +0000 UTC m=+0.019115810 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:51:16.165 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:15 smithi149 podman[77413]: 2024-03-24 11:51:15.881733757 +0000 UTC m=+0.139712670 container create 1ac3379426b317ff903ca2d1eee20ba9e815ec02ab841a811641c78232d79b2d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:51:16.165 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:16 smithi149 podman[77413]: 2024-03-24 11:51:16.082393266 +0000 UTC m=+0.340372180 container init 1ac3379426b317ff903ca2d1eee20ba9e815ec02ab841a811641c78232d79b2d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T11:51:16.165 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:16 smithi149 podman[77413]: 2024-03-24 11:51:16.085453574 +0000 UTC m=+0.343432491 container start 1ac3379426b317ff903ca2d1eee20ba9e815ec02ab841a811641c78232d79b2d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T11:51:16.165 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:16 smithi149 podman[77413]: 2024-03-24 11:51:16.165648018 +0000 UTC m=+0.423626964 container attach 1ac3379426b317ff903ca2d1eee20ba9e815ec02ab841a811641c78232d79b2d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T11:51:16.475 INFO:journalctl@ceph.osd.6.smithi149.stdout:Mar 24 11:51:16 smithi149 podman[77413]: 2024-03-24 11:51:16.373422456 +0000 UTC m=+0.631401370 container died 1ac3379426b317ff903ca2d1eee20ba9e815ec02ab841a811641c78232d79b2d (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:51:16.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:16.679+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:16.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:16.679+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:16.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:16.679+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:16.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:16 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:16.679+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:17.101 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T11:51:17.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:17.145 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T11:51:17.145 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T11:51:17.145 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.7 2024-03-24T11:51:17.472 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 systemd[1]: Stopping Ceph osd.7 for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:51:17.473 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:17.471+0000 7f02d82c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:51:17.725 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:17.471+0000 7f02d82c8700 -1 osd.7 153 *** Got signal Terminated *** 2024-03-24T11:51:17.725 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:17.471+0000 7f02d82c8700 -1 osd.7 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:51:17.725 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:17.711+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:17.725 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:17.711+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:17.725 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:17.711+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:17.725 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:17 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:17.711+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:18.974 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:18.686+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:18.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:18.686+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:18.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:18.686+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:18.975 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:18 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:18.686+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:19.456Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.145:8765: connect: connection refused" 2024-03-24T11:51:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:19.458Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.145:8765: connect: connection refused" 2024-03-24T11:51:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:19.458Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.145:8765: connect: connection refused" 2024-03-24T11:51:19.725 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:19.460Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.145:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.145:8765: connect: connection refused" 2024-03-24T11:51:20.128 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:19.734+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:20.128 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:19.734+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:20.128 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:19.734+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:20.128 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:19 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:19.734+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:21.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:20 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:20.769+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:21.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:20 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:20.769+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:21.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:20 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:20.769+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:21.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:20 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:20.769+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:22.224 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:21 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:21.736+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-24T11:50:24.096075+0000 front 2024-03-24T11:50:24.096124+0000 (oldest deadline 2024-03-24T11:50:49.396026+0000) 2024-03-24T11:51:22.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:21 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:21.736+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6814 osd.1 since back 2024-03-24T11:50:33.497199+0000 front 2024-03-24T11:50:33.497226+0000 (oldest deadline 2024-03-24T11:50:56.997140+0000) 2024-03-24T11:51:22.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:21 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:21.736+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6822 osd.2 since back 2024-03-24T11:50:42.297996+0000 front 2024-03-24T11:50:42.297906+0000 (oldest deadline 2024-03-24T11:51:07.597846+0000) 2024-03-24T11:51:22.225 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:21 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7[68386]: 2024-03-24T11:51:21.736+0000 7f02d48bc700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.145:6830 osd.3 since back 2024-03-24T11:50:47.598427+0000 front 2024-03-24T11:50:47.598364+0000 (oldest deadline 2024-03-24T11:51:11.698251+0000) 2024-03-24T11:51:23.089 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:22 smithi149 podman[77521]: 2024-03-24 11:51:22.812368423 +0000 UTC m=+5.447059005 container died f5d48891e80cbda39c86f42e17bca4901ff80b24f7e3a892bd31739eafc87296 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:51:23.475 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:23 smithi149 podman[77521]: 2024-03-24 11:51:23.188394957 +0000 UTC m=+5.823085548 container cleanup f5d48891e80cbda39c86f42e17bca4901ff80b24f7e3a892bd31739eafc87296 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T11:51:23.475 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:23 smithi149 bash[77521]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7 2024-03-24T11:51:23.802 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:23 smithi149 podman[77551]: 2024-03-24 11:51:23.487929082 +0000 UTC m=+0.670625554 container remove f5d48891e80cbda39c86f42e17bca4901ff80b24f7e3a892bd31739eafc87296 (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:51:23.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:51:23.550Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:23.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:23.551Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:23.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:23 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:23.552Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:24.109 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:23 smithi149 podman[77643]: 2024-03-24 11:51:23.70631027 +0000 UTC m=+0.019265588 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:51:24.109 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:23 smithi149 podman[77643]: 2024-03-24 11:51:23.815624441 +0000 UTC m=+0.128579790 container create a0475e47d40e2a0c29f6166772711684990f07aef5e9c106c2cd2d57f56e25de (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:51:24.109 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:24 smithi149 podman[77643]: 2024-03-24 11:51:24.026374109 +0000 UTC m=+0.339329413 container init a0475e47d40e2a0c29f6166772711684990f07aef5e9c106c2cd2d57f56e25de (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T11:51:24.110 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:24 smithi149 podman[77643]: 2024-03-24 11:51:24.029479704 +0000 UTC m=+0.342435013 container start a0475e47d40e2a0c29f6166772711684990f07aef5e9c106c2cd2d57f56e25de (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:51:24.110 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:24 smithi149 podman[77643]: 2024-03-24 11:51:24.109642345 +0000 UTC m=+0.422597658 container attach a0475e47d40e2a0c29f6166772711684990f07aef5e9c106c2cd2d57f56e25de (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:51:24.475 INFO:journalctl@ceph.osd.7.smithi149.stdout:Mar 24 11:51:24 smithi149 podman[77643]: 2024-03-24 11:51:24.314867272 +0000 UTC m=+0.627822580 container died a0475e47d40e2a0c29f6166772711684990f07aef5e9c106c2cd2d57f56e25de (image=quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T11:51:24.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=error ts=2024-03-24T11:51:24.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:24.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:24.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi145.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.145:8443: connect: connection refused" 2024-03-24T11:51:24.611 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:24 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=warn ts=2024-03-24T11:51:24.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-03-24T11:51:24.939 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T11:51:24.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:24.981 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T11:51:24.981 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T11:51:24.981 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T11:51:25.207 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 systemd[1]: Stopping Ceph prometheus.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.207Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.208Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:51:25.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.208Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-24T11:51:25.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a[53996]: ts=2024-03-24T11:51:25.208Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-24T11:51:25.476 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 podman[77721]: 2024-03-24 11:51:25.21796237 +0000 UTC m=+0.137555709 container died 595718fc2704a4f5f9efa27867c67661c07e9e207f94d16242dc11a6bff796c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:25.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 podman[77721]: 2024-03-24 11:51:25.765858488 +0000 UTC m=+0.685451823 container cleanup 595718fc2704a4f5f9efa27867c67661c07e9e207f94d16242dc11a6bff796c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:25.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 bash[77721]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:51:25.975 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 24 11:51:25 smithi149 bash[77747]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-24T11:51:26.037 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T11:51:26.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:26.074 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T11:51:26.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:51:27.861 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:27 smithi145 systemd[1]: Stopping Ceph node-exporter.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:51:27.861 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:27 smithi145 podman[92861]: 2024-03-24 11:51:27.705344423 +0000 UTC m=+0.118455062 container died 77953b6c5fa5f0c08cf66fbd4f7174f6b95360235920cc548d36f83afa388c4a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:28.238 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 podman[92861]: 2024-03-24 11:51:28.036086784 +0000 UTC m=+0.449197431 container cleanup 77953b6c5fa5f0c08cf66fbd4f7174f6b95360235920cc548d36f83afa388c4a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:28.238 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 bash[92861]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T11:51:28.238 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 podman[92875]: 2024-03-24 11:51:28.203795772 +0000 UTC m=+0.499059612 container remove 77953b6c5fa5f0c08cf66fbd4f7174f6b95360235920cc548d36f83afa388c4a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:28.238 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:51:28.548 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:28 smithi145 systemd[1]: Stopping Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:51:28.548 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 bash[92888]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-24T11:51:28.548 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T11:51:28.548 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 systemd[1]: Stopped Ceph node-exporter.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:51:28.548 INFO:journalctl@ceph.node-exporter.a.smithi145.stdout:Mar 24 11:51:28 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 4.019s CPU time. 2024-03-24T11:51:28.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:28 smithi145 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a[58983]: level=info ts=2024-03-24T11:51:28.548Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:51:28.861 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:28 smithi145 podman[92939]: 2024-03-24 11:51:28.599625859 +0000 UTC m=+0.152043720 container died e32a8e22dad4d60a93060528f3b6c6d6513623a23c5a11a7bcfc383f578f0eb4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:29.182 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:28 smithi145 podman[92939]: 2024-03-24 11:51:28.965795637 +0000 UTC m=+0.518213540 container cleanup e32a8e22dad4d60a93060528f3b6c6d6513623a23c5a11a7bcfc383f578f0eb4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:29.183 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:28 smithi145 bash[92939]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:51:29.183 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:28 smithi145 bash[92966]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-24T11:51:29.447 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:29 smithi145 podman[92953]: 2024-03-24 11:51:29.18339164 +0000 UTC m=+0.575508588 container remove e32a8e22dad4d60a93060528f3b6c6d6513623a23c5a11a7bcfc383f578f0eb4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:29.447 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:29 smithi145 podman[92953]: 2024-03-24 11:51:29.425763624 +0000 UTC m=+0.817880588 volume remove 7a54861f36cc75142d3cbdc6e672a80bf4789b1ba02e3ed1fd39814fa98706a9 2024-03-24T11:51:29.447 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:29 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:51:29.448 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:29 smithi145 systemd[1]: Stopped Ceph alertmanager.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:51:29.448 INFO:journalctl@ceph.alertmanager.a.smithi145.stdout:Mar 24 11:51:29 smithi145 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.129s CPU time. 2024-03-24T11:52:03.322 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:52:05.128 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:04 smithi149 systemd[1]: Stopping Ceph node-exporter.b for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:52:05.128 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:04 smithi149 podman[78078]: 2024-03-24 11:52:04.901134835 +0000 UTC m=+0.109333817 container died 54579c2f54d5e391ab02aa731a6a769dd07e7cd0eee132433a7d5e650da44ddd (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:52:05.474 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 podman[78078]: 2024-03-24 11:52:05.340595793 +0000 UTC m=+0.548794774 container cleanup 54579c2f54d5e391ab02aa731a6a769dd07e7cd0eee132433a7d5e650da44ddd (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:52:05.475 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 bash[78078]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T11:52:05.794 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 podman[78091]: 2024-03-24 11:52:05.575692044 +0000 UTC m=+0.675924593 container remove 54579c2f54d5e391ab02aa731a6a769dd07e7cd0eee132433a7d5e650da44ddd (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:52:05.794 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:52:05.795 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 bash[78105]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-24T11:52:05.795 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T11:52:05.795 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 systemd[1]: Stopped Ceph node-exporter.b for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:52:05.795 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 24 11:52:05 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 3.922s CPU time. 2024-03-24T11:52:06.170 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:05 smithi149 systemd[1]: Stopping Ceph grafana.a for a2bb64be-e9d1-11ee-95cd-87774f69a715... 2024-03-24T11:52:06.171 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:05 smithi149 ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a[51132]: t=2024-03-24T11:52:05+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-24T11:52:06.171 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:05 smithi149 podman[78171]: 2024-03-24 11:52:05.922740139 +0000 UTC m=+0.118846428 container died 1465ddad3f0c83203b96c92fda302639c46621dff9f6f4cd3bdd1f1981d34ab7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, release=236.1648460182, io.openshift.expose-services=, version=8.5, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-24T11:52:06.454 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:06 smithi149 podman[78171]: 2024-03-24 11:52:06.244871763 +0000 UTC m=+0.440978050 container cleanup 1465ddad3f0c83203b96c92fda302639c46621dff9f6f4cd3bdd1f1981d34ab7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, version=8.5, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.openshift.expose-services=, description=Ceph Grafana Container, io.openshift.tags=base rhel8, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, vcs-type=git) 2024-03-24T11:52:06.454 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:06 smithi149 bash[78171]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a 2024-03-24T11:52:06.454 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:06 smithi149 bash[78196]: Error: no container with name or ID "ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-24T11:52:06.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:06 smithi149 podman[78184]: 2024-03-24 11:52:06.45439752 +0000 UTC m=+0.533712044 container remove 1465ddad3f0c83203b96c92fda302639c46621dff9f6f4cd3bdd1f1981d34ab7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715-grafana-a, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182) 2024-03-24T11:52:06.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:06 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-24T11:52:06.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:06 smithi149 systemd[1]: Stopped Ceph grafana.a for a2bb64be-e9d1-11ee-95cd-87774f69a715. 2024-03-24T11:52:06.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 24 11:52:06 smithi149 systemd[1]: ceph-a2bb64be-e9d1-11ee-95cd-87774f69a715@grafana.a.service: Consumed 4.236s CPU time. 2024-03-24T11:52:29.456 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:52:29.491 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:52:29.521 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T11:52:29.523 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921/remote/smithi145/crash 2024-03-24T11:52:29.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:52:29.668 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:52:29.668 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:52:29.671 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921/remote/smithi149/crash 2024-03-24T11:52:29.672 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:52:29.713 INFO:teuthology.orchestra.run.smithi149.stderr:tar: /var/lib/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:52:29.714 INFO:teuthology.orchestra.run.smithi149.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:52:29.715 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T11:52:29.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:52:29.748 INFO:teuthology.orchestra.run.smithi145.stdout:2024-03-24T11:33:05.824924+0000 mon.a (mon.0) 477 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:52:29.748 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T11:52:29.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[SEC\]' /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:52:29.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[ERR\]' /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:52:29.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[WRN\]' /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:52:29.956 INFO:teuthology.orchestra.run.smithi145.stdout:2024-03-24T11:33:05.824924+0000 mon.a (mon.0) 477 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:52:29.956 INFO:tasks.cephadm:Compressing logs... 2024-03-24T11:52:29.956 DEBUG:teuthology.orchestra.run.smithi145:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:52:29.998 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:52:30.023 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:52:30.023 INFO:teuthology.orchestra.run.smithi145.stderr:: No such file or directory 2024-03-24T11:52:30.023 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T11:52:30.023 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.smpl.smithi145.wrtpsu.log 2024-03-24T11:52:30.023 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi145.rqljdz.log 2024-03-24T11:52:30.023 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.smpl.smithi145.wrtpsu.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:52:30.025 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi145.rqljdz.log: 76.8% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.smpl.smithi145.wrtpsu.log.gz 2024-03-24T11:52:30.025 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T11:52:30.025 INFO:teuthology.orchestra.run.smithi145.stderr: 76.8% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi145.rqljdz.log.gz 2024-03-24T11:52:30.025 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T11:52:30.025 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T11:52:30.025 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-24T11:52:30.026 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T11:52:30.026 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/tcmu-runner.log: 84.6% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-24T11:52:30.026 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:52:30.027 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:52:30.027 INFO:teuthology.orchestra.run.smithi149.stderr:: No such file or directory 2024-03-24T11:52:30.027 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T11:52:30.028 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:52:30.028 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.4.log: /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi149.gflbhe.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.smpl.smithi149.djutzp.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi149.gflbhe.log: /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.smpl.smithi149.djutzp.log: 76.8% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi149.gflbhe.log.gz 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr: 76.2% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-client.rgw.smpl.smithi149.djutzp.log.gz 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.cephadm.log: 90.6% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr: 85.9% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:52:30.032 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T11:52:30.034 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-volume.log: 87.8% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:52:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.0.log: 94.2% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:52:30.034 INFO:teuthology.orchestra.run.smithi145.stderr: 90.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:52:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:52:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T11:52:30.035 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.b.log: 88.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:52:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log: 91.7% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:52:30.036 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:52:30.039 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.2.log: 93.1% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:52:30.039 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T11:52:30.044 INFO:teuthology.orchestra.run.smithi149.stderr: 91.4% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:52:30.064 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.2% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:52:30.069 INFO:teuthology.orchestra.run.smithi149.stderr: 89.0% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T11:52:30.335 INFO:teuthology.orchestra.run.smithi149.stderr: 92.4% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T11:52:30.381 INFO:teuthology.orchestra.run.smithi145.stderr: 88.6% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T11:52:30.533 INFO:teuthology.orchestra.run.smithi145.stderr: 92.1% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T11:52:30.997 INFO:teuthology.orchestra.run.smithi145.stderr: 91.0% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T11:52:31.095 INFO:teuthology.orchestra.run.smithi149.stderr: 93.3% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T11:52:31.326 INFO:teuthology.orchestra.run.smithi145.stderr: 93.3% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T11:52:31.416 INFO:teuthology.orchestra.run.smithi149.stderr: 93.5% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T11:52:31.437 INFO:teuthology.orchestra.run.smithi145.stderr: 93.7% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T11:52:31.454 INFO:teuthology.orchestra.run.smithi149.stderr: 93.8% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T11:52:31.552 INFO:teuthology.orchestra.run.smithi149.stderr: 93.7% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T11:52:31.554 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-03-24T11:52:31.554 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m1.542s 2024-03-24T11:52:31.554 INFO:teuthology.orchestra.run.smithi149.stderr:user 0m5.586s 2024-03-24T11:52:31.554 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m0.227s 2024-03-24T11:52:31.664 INFO:teuthology.orchestra.run.smithi145.stderr: 93.7% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T11:52:31.856 INFO:teuthology.orchestra.run.smithi145.stderr: 93.7% -- replaced with /var/log/ceph/a2bb64be-e9d1-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T11:52:31.859 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-24T11:52:31.859 INFO:teuthology.orchestra.run.smithi145.stderr:real 0m1.849s 2024-03-24T11:52:31.859 INFO:teuthology.orchestra.run.smithi145.stderr:user 0m7.722s 2024-03-24T11:52:31.859 INFO:teuthology.orchestra.run.smithi145.stderr:sys 0m0.290s 2024-03-24T11:52:31.859 INFO:tasks.cephadm:Archiving logs... 2024-03-24T11:52:31.859 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921/remote/smithi145/log 2024-03-24T11:52:31.861 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:52:32.849 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921/remote/smithi149/log 2024-03-24T11:52:32.850 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:52:33.675 INFO:tasks.cephadm:Removing cluster... 2024-03-24T11:52:33.675 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 --force 2024-03-24T11:52:33.966 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a2bb64be-e9d1-11ee-95cd-87774f69a715 --force 2024-03-24T11:52:34.249 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T11:52:34.249 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:52:34.264 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:52:34.278 INFO:tasks.cephadm:Teardown complete 2024-03-24T11:52:34.279 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T11:52:34.288 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T11:52:34.289 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:52:34.307 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:52:34.318 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:52:34.323 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:52:34.323 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-24T11:52:34.323 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +932us[ +930us] +/- 82ms 2024-03-24T11:52:34.324 INFO:teuthology.orchestra.run.smithi145.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 +107us[ +105us] +/- 62ms 2024-03-24T11:52:34.324 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -812us[ -812us] +/- 66ms 2024-03-24T11:52:34.324 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:52:34.332 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:52:34.338 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:52:34.338 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-03-24T11:52:34.338 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +909us[ +909us] +/- 82ms 2024-03-24T11:52:34.338 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 7 377 82 +106us[ +115us] +/- 61ms 2024-03-24T11:52:34.338 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -842us[ -842us] +/- 66ms 2024-03-24T11:52:34.338 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:52:34.338 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T11:52:34.347 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T11:52:34.348 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T11:52:34.357 DEBUG:teuthology.orchestra.run.smithi145:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:52:34.444 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:52:34.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:52:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:52:34.552 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:52:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:52:34.582 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T11:52:34.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T11:52:34.600 INFO:teuthology.task.internal:Duration was 1701.190709 seconds 2024-03-24T11:52:34.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T11:52:34.610 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T11:52:34.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:52:34.612 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:52:34.653 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:52:34.664 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:52:35.144 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T11:52:35.144 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-03-24T11:52:35.145 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:52:35.205 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-03-24T11:52:35.205 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:52:35.231 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T11:52:35.232 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:52:35.247 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:52:35.307 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T11:52:35.307 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:52:35.316 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:52:35.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T11:52:35.747 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T11:52:35.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:52:35.774 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:52:35.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T11:52:35.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:52:35.816 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:52:35.839 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-03-24T11:52:35.866 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-03-24T11:52:35.886 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:52:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:52:35.913 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:52:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:52:35.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T11:52:35.950 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T11:52:35.951 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921/remote/smithi145 2024-03-24T11:52:35.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:52:35.999 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921/remote/smithi149 2024-03-24T11:52:35.999 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:52:36.040 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T11:52:36.040 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:52:36.042 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:52:36.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T11:52:36.097 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T11:52:36.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T11:52:36.106 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T11:52:36.107 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:52:36.109 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:52:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:52 /home/ubuntu/cephtest 2024-03-24T11:52:36.142 INFO:teuthology.orchestra.run.smithi149.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:52 /home/ubuntu/cephtest 2024-03-24T11:52:36.143 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T11:52:36.152 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T11:52:36.172 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T11:52:36.201 DEBUG:teuthology.nuke:shortname: smithi145 2024-03-24T11:52:36.201 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:52:36.226 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:14:12.602723', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwm6wqBFZn2UGMeaF0VzicBXxM770gis4JyqHWgkTzsetR6P4i6XzU2gZ24EJurcNCDSUNkyLuGL4k3bqBqBv4='} 2024-03-24T11:52:36.229 DEBUG:teuthology.nuke:shortname: smithi149 2024-03-24T11:52:36.229 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:52:36.251 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617921', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:14:12.600968', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDlHgECTRqX4sRlkcUa7B1Dst/KYJZg6YqRHj/fr8QWzXPf0nBFRBaEVO9zCS0ft+pjxZw8Uf+xe1owZwW2Ikrw='} 2024-03-24T11:52:36.278 INFO:teuthology.orchestra.console.smithi145:Power off 2024-03-24T11:52:36.278 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:52:36.302 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-03-24T11:52:36.302 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:52:36.318 INFO:teuthology.orchestra.console.smithi149:Power off 2024-03-24T11:52:36.318 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:52:36.327 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-24T11:52:36.340 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2024-03-24T11:52:36.340 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:52:36.360 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-03-24T11:52:40.329 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:52:40.361 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:52:40.440 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-24T11:52:40.470 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-03-24T11:52:44.442 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:52:44.471 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:52:44.552 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-24T11:52:44.580 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2024-03-24T11:52:44.680 INFO:teuthology.orchestra.console.smithi149:Power off completed 2024-03-24T11:52:48.554 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:52:48.663 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-24T11:52:48.764 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-03-24T11:52:48.868 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 1701.1907088756561 failure_reason: '"2024-03-24T11:33:05.824924+0000 mon.a (mon.0) 477 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-24T11:52:48.868 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:52:48.947 INFO:teuthology.run:FAIL