2024-05-06T12:18:44.214 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-06T12:18:44.214 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140 --verbose --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --name teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140/orig.config.yaml 2024-05-06T12:18:44.243 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T12:18:44.326 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7688140' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa workunit: branch: squid sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 3735 sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZcHTy/FY5NXQv9ysBe429FHFo9gVXuEan7BlMQMgErhgvxAcQ//sJ8UWbs1rSK6GSjqhfwl3FhRURWczZxpXw= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCzZsL+oNnrbNXuhRYF0rXko3EBSy0fL6JAoBBguEnSRJPXqn0x2cnvFqh50dPIflyLEVyQ43z2VULDdSM45WLY= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-03_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-06T12:18:44.326 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa; will attempt to use it 2024-05-06T12:18:44.327 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks 2024-05-06T12:18:44.327 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-06T12:18:44.328 INFO:teuthology.task.internal:Checking packages... 2024-05-06T12:18:44.351 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '70ce9539b62a15b9de429c1f0e91a1e012b099aa' 2024-05-06T12:18:44.351 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-06T12:18:44.351 INFO:teuthology.packaging:ref: None 2024-05-06T12:18:44.351 INFO:teuthology.packaging:tag: None 2024-05-06T12:18:44.351 INFO:teuthology.packaging:branch: squid 2024-05-06T12:18:44.351 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:18:44.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-06T12:18:44.574 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2504.gb05eb275 2024-05-06T12:18:44.575 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-06T12:18:44.583 INFO:teuthology.task.internal:no buildpackages task found 2024-05-06T12:18:44.584 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-06T12:18:44.592 INFO:teuthology.task.internal:Saving configuration 2024-05-06T12:18:44.604 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-06T12:18:44.612 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T12:18:44.636 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 12:13:35.995517', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZcHTy/FY5NXQv9ysBe429FHFo9gVXuEan7BlMQMgErhgvxAcQ//sJ8UWbs1rSK6GSjqhfwl3FhRURWczZxpXw='} 2024-05-06T12:18:44.654 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 12:13:35.997019', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCzZsL+oNnrbNXuhRYF0rXko3EBSy0fL6JAoBBguEnSRJPXqn0x2cnvFqh50dPIflyLEVyQ43z2VULDdSM45WLY='} 2024-05-06T12:18:44.654 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-06T12:18:44.664 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-05-06T12:18:44.664 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-05-06T12:18:44.664 INFO:teuthology.run_tasks:Running task console_log... 2024-05-06T12:18:44.737 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f869350d0>, signals=[15]) 2024-05-06T12:18:44.738 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-06T12:18:44.746 INFO:teuthology.task.internal:Opening connections... 2024-05-06T12:18:44.746 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-05-06T12:18:44.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T12:18:44.824 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-05-06T12:18:44.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T12:18:44.897 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-06T12:18:44.911 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-05-06T12:18:44.930 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-05-06T12:18:44.930 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-05-06T12:18:44.988 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-06T12:18:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-06T12:18:44.990 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-05-06T12:18:45.013 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-05-06T12:18:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-05-06T12:18:45.031 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-05-06T12:18:45.088 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-06T12:18:45.089 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-06T12:18:45.089 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-06T12:18:45.089 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-05-06T12:18:45.112 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-06T12:18:45.123 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-06T12:18:45.131 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-06T12:18:45.131 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-05-06T12:18:45.134 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-05-06T12:18:45.150 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-06T12:18:45.158 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-06T12:18:45.158 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-05-06T12:18:45.191 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-05-06T12:18:45.290 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-06T12:18:45.343 INFO:teuthology.run_tasks:Running task kernel... 2024-05-06T12:18:45.363 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-06T12:18:45.364 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-06T12:18:45.364 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T12:18:45.364 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T12:18:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:18:45.382 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-05-06T12:18:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:18:45.383 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-05-06T12:18:45.439 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-437.el9.x86_64 2024-05-06T12:18:45.439 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-437.el9.x86_64 2024-05-06T12:18:45.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-05-06T12:18:45.441 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-437.el9.x86_64 2024-05-06T12:18:45.441 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-437.el9.x86_64 2024-05-06T12:18:45.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-05-06T12:18:47.875 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-05-06T12:18:48.838 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-05-06T12:18:50.426 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-05-06T12:18:50.839 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-05-06T12:18:56.065 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.2 MB 00:01 2024-05-06T12:18:56.626 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.2 MB 00:02 2024-05-06T12:18:57.757 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-05-06T12:18:58.477 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-05-06T12:18:59.594 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-06T12:19:00.130 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-06T12:19:04.975 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-06T12:19:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-05-06T12:19:06.973 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-06T12:19:06.973 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-06T12:19:07.034 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-06T12:19:07.034 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-06T12:19:07.064 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-06T12:19:07.066 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-05-06T12:19:07.066 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-06T12:19:07.126 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-05-06T12:19:07.127 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-05-06T12:19:07.127 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-05-06T12:19:07.183 DEBUG:teuthology.orchestra.run.smithi067:> echo no | sudo yum reinstall kernel || true 2024-05-06T12:19:07.235 DEBUG:teuthology.orchestra.run.smithi086:> echo no | sudo yum reinstall kernel || true 2024-05-06T12:19:07.728 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:02 ago on Mon 06 May 2024 12:19:05 PM UTC. 2024-05-06T12:19:07.744 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 12:19:04 PM UTC. 2024-05-06T12:19:07.847 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:Reinstalling: 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 255 k 2024-05-06T12:19:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 0 2024-05-06T12:19:07.870 INFO:teuthology.orchestra.run.smithi086.stderr:Operation aborted. 2024-05-06T12:19:07.870 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-06T12:19:07.884 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-05-06T12:19:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:Operation aborted. 2024-05-06T12:19:07.934 INFO:teuthology.orchestra.run.smithi086.stdout:Is this ok [y/N]: 2024-05-06T12:19:07.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum reinstall -y kernel || true 2024-05-06T12:19:07.943 INFO:teuthology.orchestra.run.smithi067.stdout:Is this ok [y/N]: 2024-05-06T12:19:07.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum reinstall -y kernel || true 2024-05-06T12:19:08.424 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 12:19:05 PM UTC. 2024-05-06T12:19:08.465 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:04 ago on Mon 06 May 2024 12:19:04 PM UTC. 2024-05-06T12:19:08.540 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-05-06T12:19:08.540 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout:Reinstalling: 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 255 k 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 0 2024-05-06T12:19:08.541 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-05-06T12:19:08.586 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-05-06T12:19:08.587 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-05-06T12:19:08.797 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.2 MB/s | 255 kB 00:00 2024-05-06T12:19:08.797 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-05-06T12:19:08.797 INFO:teuthology.orchestra.run.smithi086.stdout:Total 997 kB/s | 255 kB 00:00 2024-05-06T12:19:08.814 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-05-06T12:19:08.976 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-05-06T12:19:08.977 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-05-06T12:19:08.981 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-05-06T12:19:08.982 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-05-06T12:19:09.123 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-05-06T12:19:09.206 INFO:teuthology.orchestra.run.smithi086.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T12:19:09.206 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T12:19:09.469 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64.rpm 302 kB/s | 255 kB 00:00 2024-05-06T12:19:09.469 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-06T12:19:09.469 INFO:teuthology.orchestra.run.smithi067.stdout:Total 289 kB/s | 255 kB 00:00 2024-05-06T12:19:09.492 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-05-06T12:19:09.683 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-05-06T12:19:09.684 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-05-06T12:19:09.689 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-05-06T12:19:09.689 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-05-06T12:19:09.835 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-05-06T12:19:09.882 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T12:19:09.883 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T12:19:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T12:19:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T12:19:10.543 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T12:19:10.543 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:19:10.543 INFO:teuthology.orchestra.run.smithi086.stdout:Reinstalled: 2024-05-06T12:19:10.543 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-06T12:19:10.544 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:19:10.544 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-05-06T12:19:10.560 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T12:19:10.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T12:19:10.717 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-05-06T12:19:10.763 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-06T12:19:10.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-06T12:19:10.763 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-06T12:19:10.763 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-06T12:19:10.763 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T12:19:10.795 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-05-06T12:19:10.822 DEBUG:teuthology.parallel:result is None 2024-05-06T12:19:11.220 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T12:19:11.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:19:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalled: 2024-05-06T12:19:11.221 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-06T12:19:11.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:19:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-06T12:19:11.390 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-05-06T12:19:11.438 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-06T12:19:11.439 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-06T12:19:11.439 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-06T12:19:11.439 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-06T12:19:11.439 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T12:19:11.471 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-05-06T12:19:11.499 DEBUG:teuthology.parallel:result is None 2024-05-06T12:19:11.500 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-06T12:19:11.511 INFO:teuthology.task.internal:Creating test directory... 2024-05-06T12:19:11.512 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T12:19:11.543 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T12:19:11.559 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-06T12:19:11.566 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-06T12:19:11.572 INFO:teuthology.task.internal:Creating archive directory... 2024-05-06T12:19:11.573 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T12:19:11.599 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T12:19:11.627 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-06T12:19:11.634 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-06T12:19:11.634 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T12:19:11.658 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T12:19:11.698 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T12:19:11.700 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T12:19:11.708 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T12:19:11.710 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T12:19:11.711 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-06T12:19:11.718 INFO:teuthology.task.internal:Configuring sudo... 2024-05-06T12:19:11.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T12:19:11.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T12:19:11.778 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-06T12:19:11.788 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-06T12:19:11.789 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T12:19:11.817 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T12:19:11.833 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T12:19:11.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T12:19:11.956 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T12:19:12.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T12:19:12.078 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:19:12.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T12:19:12.142 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T12:19:12.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T12:19:12.231 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T12:19:12.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T12:19:12.352 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-06T12:19:12.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T12:19:12.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-05-06T12:19:12.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-05-06T12:19:12.460 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T12:19:12.495 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T12:19:12.905 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-06T12:19:12.913 INFO:teuthology.task.internal:Starting timer... 2024-05-06T12:19:12.913 INFO:teuthology.run_tasks:Running task pcp... 2024-05-06T12:19:12.928 INFO:teuthology.run_tasks:Running task selinux... 2024-05-06T12:19:12.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-05-06T12:19:13.008 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-05-06T12:19:13.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-05-06T12:19:13.047 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-05-06T12:19:13.049 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-06T12:19:13.049 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-05-06T12:19:13.069 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-05-06T12:19:13.070 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-05-06T12:19:13.120 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-05-06T12:19:13.120 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-05-06T12:19:13.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T12:19:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:19:13.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T12:19:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:19:13.184 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-06T12:19:13.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-05-06T12:19:13.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-05-06T12:19:13.248 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-06T12:19:13.259 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-06T12:19:13.276 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-06T12:19:13.278 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-06T12:23:59.563 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-05-06T12:23:59.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-06T12:23:59.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T12:23:59.644 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-05-06T12:23:59.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-06T12:23:59.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-05-06T12:23:59.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T12:23:59.799 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-05-06T12:23:59.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-05-06T12:23:59.878 INFO:teuthology.run_tasks:Running task clock... 2024-05-06T12:23:59.888 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-06T12:23:59.888 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T12:23:59.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T12:23:59.891 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T12:23:59.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T12:23:59.918 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-06T12:23:59.932 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-06T12:23:59.946 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-06T12:23:59.956 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-05-06T12:23:59.959 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-06T12:23:59.967 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-05-06T12:23:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-06T12:23:59.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-05-06T12:23:59.991 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-05-06T12:23:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-06T12:24:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-06T12:24:00.018 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-06T12:24:00.045 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-06T12:24:00.049 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T12:24:00.049 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-06T12:24:00.049 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.049 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.049 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.049 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.073 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-05-06T12:24:00.077 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T12:24:00.077 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-05-06T12:24:00.077 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.077 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.077 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.077 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:24:00.077 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-06T12:24:00.202 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa'} 2024-05-06T12:24:00.203 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-05-06T12:24:00.203 INFO:tasks.cephadm:Cluster fsid is 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:00.203 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-06T12:24:00.204 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.67', 'mon.c': '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]', 'mon.b': '172.21.15.86'} 2024-05-06T12:24:00.204 INFO:tasks.cephadm:First mon is mon.a on smithi067 2024-05-06T12:24:00.204 INFO:tasks.cephadm:First mgr is y 2024-05-06T12:24:00.204 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-06T12:24:00.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-05-06T12:24:00.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-05-06T12:24:00.254 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-05-06T12:24:00.255 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-06T12:24:00.523 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 6 12:24 /home/ubuntu/cephtest/cephadm 2024-05-06T12:24:00.523 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-06T12:24:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 6 12:24 /home/ubuntu/cephtest/cephadm 2024-05-06T12:24:00.685 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-06T12:24:00.702 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-06T12:24:00.728 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-05-06T12:24:00.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-06T12:24:00.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-06T12:24:02.348 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-06T12:24:02.440 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-05-06T12:24:23.551 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-05-06T12:24:25.414 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:24:25.414 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-06T12:24:25.414 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-06T12:24:25.414 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-05-06T12:24:25.414 INFO:teuthology.orchestra.run.smithi067.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-06T12:24:25.415 INFO:teuthology.orchestra.run.smithi067.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-06T12:24:25.415 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-06T12:24:25.415 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:24:25.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-05-06T12:24:25.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-05-06T12:24:25.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-05-06T12:24:25.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-05-06T12:24:25.570 INFO:tasks.cephadm:Writing seed config... 2024-05-06T12:24:25.571 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-06T12:24:25.572 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-06T12:24:25.572 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:24:25.573 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-06T12:24:25.590 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 847b865c-0ba3-11ef-bc97-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-05-06T12:24:25.591 DEBUG:teuthology.orchestra.run.smithi067:mon.a> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service 2024-05-06T12:24:25.633 DEBUG:teuthology.orchestra.run.smithi067:mgr.y> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service 2024-05-06T12:24:25.675 INFO:tasks.cephadm:Bootstrapping... 2024-05-06T12:24:25.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.67 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-06T12:24:25.875 INFO:teuthology.orchestra.run.smithi067.stderr:-------------------------------------------------------------------------------- 2024-05-06T12:24:25.875 INFO:teuthology.orchestra.run.smithi067.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '847b865c-0ba3-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.67', '--skip-admin-label'] 2024-05-06T12:24:25.899 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: 4.9.4-dev 2024-05-06T12:24:25.901 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying podman|docker is present... 2024-05-06T12:24:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: 4.9.4-dev 2024-05-06T12:24:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying lvm2 is present... 2024-05-06T12:24:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying time synchronization is in place... 2024-05-06T12:24:25.932 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-06T12:24:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-05-06T12:24:25.945 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: enabled 2024-05-06T12:24:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: active 2024-05-06T12:24:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:Unit chronyd.service is enabled and running 2024-05-06T12:24:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:Repeating the final host check... 2024-05-06T12:24:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: 4.9.4-dev 2024-05-06T12:24:25.975 INFO:teuthology.orchestra.run.smithi067.stderr:podman (/bin/podman) version 4.9.4 is present 2024-05-06T12:24:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl is present 2024-05-06T12:24:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:lvcreate is present 2024-05-06T12:24:25.982 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-06T12:24:25.988 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-05-06T12:24:25.995 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: enabled 2024-05-06T12:24:26.001 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: active 2024-05-06T12:24:26.002 INFO:teuthology.orchestra.run.smithi067.stderr:Unit chronyd.service is enabled and running 2024-05-06T12:24:26.002 INFO:teuthology.orchestra.run.smithi067.stderr:Host looks OK 2024-05-06T12:24:26.002 INFO:teuthology.orchestra.run.smithi067.stderr:Cluster fsid: 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:26.002 INFO:teuthology.orchestra.run.smithi067.stderr:Acquiring lock 140163369434992 on /run/cephadm/847b865c-0ba3-11ef-bc97-c7b262605968.lock 2024-05-06T12:24:26.002 INFO:teuthology.orchestra.run.smithi067.stderr:Lock 140163369434992 acquired on /run/cephadm/847b865c-0ba3-11ef-bc97-c7b262605968.lock 2024-05-06T12:24:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying IP 172.21.15.67 port 3300 ... 2024-05-06T12:24:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying IP 172.21.15.67 port 6789 ... 2024-05-06T12:24:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:Base mon IP is 172.21.15.67, final addrv is [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-05-06T12:24:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.67 metric 100 2024-05-06T12:24:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.67 metric 100 2024-05-06T12:24:26.008 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-05-06T12:24:26.008 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-05-06T12:24:26.008 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-05-06T12:24:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-06T12:24:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-05-06T12:24:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-06T12:24:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-06T12:24:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: inet6 fe80::b737:4306:2723:e0c5/64 scope link noprefixroute 2024-05-06T12:24:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-06T12:24:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-05-06T12:24:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-06T12:24:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-06T12:24:26.127 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-05-06T12:24:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Getting image source signatures 2024-05-06T12:24:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-05-06T12:24:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-05-06T12:24:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-05-06T12:24:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-05-06T12:24:26.492 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-05-06T12:24:26.492 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-06T12:24:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Writing manifest to image destination 2024-05-06T12:24:26.770 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-06T12:24:27.176 INFO:teuthology.orchestra.run.smithi067.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-06T12:24:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-06T12:24:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:Extracting ceph user uid/gid from container image... 2024-05-06T12:24:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:stat: 167 167 2024-05-06T12:24:28.725 INFO:teuthology.orchestra.run.smithi067.stderr:Creating initial keys... 2024-05-06T12:24:29.126 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-authtool: AQD9yzhmfWWJAhAARU5KSd3h0YKG8A2hECjzMQ== 2024-05-06T12:24:30.032 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-authtool: AQD9yzhmQ4T6NxAA9+rQWi2R1Hj8UIQyoHj24g== 2024-05-06T12:24:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-authtool: AQD+yzhm4wjjLxAA4hjyrcTiXF2J41iQChQIMA== 2024-05-06T12:24:31.395 INFO:teuthology.orchestra.run.smithi067.stderr:Creating initial monmap... 2024-05-06T12:24:31.786 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-06T12:24:31.786 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-05-06T12:24:31.786 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:31.786 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-06T12:24:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:monmaptool for a [v2:172.21.15.67:3300,v1:172.21.15.67:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-06T12:24:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:setting min_mon_release = octopus 2024-05-06T12:24:32.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: set fsid to 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:32.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-06T12:24:32.267 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:24:32.267 INFO:teuthology.orchestra.run.smithi067.stderr:Creating mon... 2024-05-06T12:24:33.198 INFO:teuthology.orchestra.run.smithi067.stderr:create mon.a on 2024-05-06T12:24:33.411 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-06T12:24:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-847b865c-0ba3-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-847b865c-0ba3-11ef-bc97-c7b262605968.target. 2024-05-06T12:24:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-847b865c-0ba3-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-847b865c-0ba3-11ef-bc97-c7b262605968.target. 2024-05-06T12:24:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to reset failed state of unit ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service: Unit ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service not loaded. 2024-05-06T12:24:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/ceph-847b865c-0ba3-11ef-bc97-c7b262605968.target.wants/ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service → /etc/systemd/system/ceph-847b865c-0ba3-11ef-bc97-c7b262605968@.service. 2024-05-06T12:24:34.607 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:34 smithi067 ceph-mon[22063]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T12:24:34.607 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:34 smithi067 bash[22044]: 5e6a4bc641931bf98c6310d91ed6b832fd7f474916f920d1524d8e94ebb17342 2024-05-06T12:24:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-05-06T12:24:34.627 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-05-06T12:24:34.628 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-05-06T12:24:34.628 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to enable service . firewalld.service is not available 2024-05-06T12:24:34.628 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mon to start... 2024-05-06T12:24:34.628 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mon... 2024-05-06T12:24:34.866 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:34 smithi067 systemd[1]: Started Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:24:35.167 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: cluster: 2024-05-06T12:24:35.167 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: id: 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: services: 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.599007s) 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mgr: no daemons active 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: data: 2024-05-06T12:24:35.168 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-05-06T12:24:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-05-06T12:24:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-05-06T12:24:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: pgs: 2024-05-06T12:24:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:35.666 INFO:teuthology.orchestra.run.smithi067.stderr:mon is available 2024-05-06T12:24:35.667 INFO:teuthology.orchestra.run.smithi067.stderr:Assimilating anything we can from ceph.conf... 2024-05-06T12:24:35.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:35 smithi067 ceph-mon[22063]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T12:24:35.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:35 smithi067 ceph-mon[22063]: monmap e1: 1 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} 2024-05-06T12:24:35.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:35 smithi067 ceph-mon[22063]: fsmap 2024-05-06T12:24:35.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:35 smithi067 ceph-mon[22063]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T12:24:35.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:35 smithi067 ceph-mon[22063]: mgrmap e1: no daemons active 2024-05-06T12:24:35.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:35 smithi067 ceph-mon[22063]: from='client.? 172.21.15.67:0/4037356487' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T12:24:36.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:36.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [global] 2024-05-06T12:24:36.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: fsid = 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:36.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-05-06T12:24:36.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [mgr] 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [osd] 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-06T12:24:36.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-06T12:24:36.656 INFO:teuthology.orchestra.run.smithi067.stderr:Generating new minimal ceph.conf... 2024-05-06T12:24:36.871 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:36 smithi067 ceph-mon[22063]: from='client.? 172.21.15.67:0/4167164803' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T12:24:36.871 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:36 smithi067 ceph-mon[22063]: from='client.? 172.21.15.67:0/4167164803' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-06T12:24:37.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:37 smithi067 ceph-mon[22063]: from='client.? 172.21.15.67:0/483494461' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:24:37.746 INFO:teuthology.orchestra.run.smithi067.stderr:Restarting the monitor... 2024-05-06T12:24:38.104 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:37 smithi067 systemd[1]: Stopping Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:24:38.104 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:37 smithi067 bash[22282]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-06T12:24:38.105 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:37 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a[22059]: 2024-05-06T12:24:37.928+0000 7f75b938a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:24:38.105 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:37 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a[22059]: 2024-05-06T12:24:37.928+0000 7f75b938a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-06T12:24:38.355 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 podman[22295]: 2024-05-06 12:24:38.103259181 +0000 UTC m=+0.289845958 container died 5e6a4bc641931bf98c6310d91ed6b832fd7f474916f920d1524d8e94ebb17342 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, release=754, ceph=True, name=centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream) 2024-05-06T12:24:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 podman[22295]: 2024-05-06 12:24:38.411705968 +0000 UTC m=+0.598292743 container cleanup 5e6a4bc641931bf98c6310d91ed6b832fd7f474916f920d1524d8e94ebb17342 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, architecture=x86_64, release=754, name=centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.component=centos-stream-container) 2024-05-06T12:24:38.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 bash[22295]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a 2024-05-06T12:24:38.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 bash[22322]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-06T12:24:38.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 podman[22309]: 2024-05-06 12:24:38.587436048 +0000 UTC m=+0.478106408 container remove 5e6a4bc641931bf98c6310d91ed6b832fd7f474916f920d1524d8e94ebb17342 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, release=754, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8) 2024-05-06T12:24:38.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-06T12:24:38.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 systemd[1]: Stopped Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:24:38.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 systemd[1]: Starting Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:24:39.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 podman[22393]: 2024-05-06 12:24:38.734640242 +0000 UTC m=+0.019860990 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-05-06T12:24:39.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:38 smithi067 podman[22393]: 2024-05-06 12:24:38.838458953 +0000 UTC m=+0.123679737 container create f5b02626cc9d13f6a751addaf5eeb80b78c29865215e5d9e4a9cad98b5889c17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, version=8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-type=git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0) 2024-05-06T12:24:39.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 podman[22393]: 2024-05-06 12:24:39.048928982 +0000 UTC m=+0.334149728 container init f5b02626cc9d13f6a751addaf5eeb80b78c29865215e5d9e4a9cad98b5889c17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, name=centos-stream, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, RELEASE=HEAD) 2024-05-06T12:24:39.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 podman[22393]: 2024-05-06 12:24:39.051905861 +0000 UTC m=+0.337126609 container start f5b02626cc9d13f6a751addaf5eeb80b78c29865215e5d9e4a9cad98b5889c17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.display-name=CentOS Stream 8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, RELEASE=HEAD, io.openshift.expose-services=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, release=754, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-06T12:24:39.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T12:24:39.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: pidfile_write: ignore empty --pid-file 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: load: jerasure load: lrc 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: RocksDB version: 6.15.5 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Compile date Apr 18 2022 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: DB SUMMARY 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: DB Session ID: KDACTD4QE94DNEEA57YV 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: CURRENT file: CURRENT 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82631 ; 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.error_if_exists: 0 2024-05-06T12:24:39.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.create_if_missing: 0 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.paranoid_checks: 1 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.env: 0x562a11ac6860 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.fs: Posix File System 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.info_log: 0x562a142c1dc0 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.statistics: (nil) 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.use_fsync: 0 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_log_file_size: 0 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.allow_fallocate: 1 2024-05-06T12:24:39.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.use_direct_reads: 0 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.db_log_dir: 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T12:24:39.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.write_buffer_manager: 0x562a143b22a0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.unordered_write: 0 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T12:24:39.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T12:24:39.087 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.row_cache: None 2024-05-06T12:24:39.087 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.wal_filter: None 2024-05-06T12:24:39.087 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T12:24:39.087 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T12:24:39.087 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.preserve_deletes: 0 2024-05-06T12:24:39.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.two_write_queues: 0 2024-05-06T12:24:39.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T12:24:39.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.atomic_flush: 0 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.log_readahead_size: 0 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T12:24:39.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_background_jobs: 2 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_background_compactions: -1 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_subcompactions: 1 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_open_files: -1 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T12:24:39.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_background_flushes: -1 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Compression algorithms supported: 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kZSTD supported: 0 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kXpressCompression supported: 0 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kLZ4Compression supported: 1 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kBZip2Compression supported: 0 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kZlibCompression supported: 1 2024-05-06T12:24:39.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: kSnappyCompression supported: 1 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.merge_operator: 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_filter: None 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562a1428dd00) 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-05-06T12:24:39.092 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_type: 0 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_index_type: 0 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_shortening: 1 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: hash_index_allow_collision: 1 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: checksum: 1 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: no_block_cache: 0 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache: 0x562a142f8170 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_options: 2024-05-06T12:24:39.093 INFO:journalctl@ceph.mon.a.smithi067.stdout: capacity : 536870912 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_shard_bits : 4 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: strict_capacity_limit : 0 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_compressed: (nil) 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: persistent_cache: (nil) 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size: 4096 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size_deviation: 10 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_restart_interval: 16 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_block_restart_interval: 1 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: metadata_block_size: 4096 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: partition_filters: 0 2024-05-06T12:24:39.094 INFO:journalctl@ceph.mon.a.smithi067.stdout: use_delta_encoding: 1 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout: whole_key_filtering: 1 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout: verify_compression: 0 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout: format_version: 4 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_index_compression: 1 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_align: 0 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression: NoCompression 2024-05-06T12:24:39.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.num_levels: 7 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:24:39.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T12:24:39.097 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T12:24:39.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.arena_block_size: 4194304 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T12:24:39.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.table_properties_collectors: 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.inplace_update_support: 0 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T12:24:39.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.bloom_locality: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.max_successive_merges: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.ttl: 2592000 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.enable_blob_files: false 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.min_blob_size: 0 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T12:24:39.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714998279078025, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714998279078792, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79181, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77444, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71753, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1714998279, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a1b807f9-3aff-4cdc-8971-efb529c25cbf", "db_session_id": "KDACTD4QE94DNEEA57YV"}} 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714998279079122, "job": 1, "event": "recovery_finished"} 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x562a142dea80 2024-05-06T12:24:39.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: DB pointer 0x562a142ee000 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** DB Stats ** 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:24:39.103 INFO:journalctl@ceph.mon.a.smithi067.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.5 0.00 0.00 1 0.000 0 0 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.5 0.00 0.00 1 0.000 0 0 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.5 0.00 0.00 1 0.000 0 0 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.5 0.00 0.00 1 0.000 0 0 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative compaction: 0.00 GB write, 25.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval compaction: 0.00 GB write, 25.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:24:39.104 INFO:journalctl@ceph.mon.a.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.5 0.00 0.00 1 0.000 0 0 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.5 0.00 0.00 1 0.000 0 0 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:24:39.105 INFO:journalctl@ceph.mon.a.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.5 0.00 0.00 1 0.000 0 0 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative compaction: 0.00 GB write, 24.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T12:24:39.106 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: starting mon.a rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???) e1 preinit fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???).mds e1 new map 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???).mds e1 print_map 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout: e1 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout: legacy client fscid: -1 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout: No filesystems configured 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T12:24:39.107 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expand map: {default=false} 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta from 'false' to 'false' 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expanded map: {default=false} 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expand map: {default=info} 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta from 'info' to 'info' 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expanded map: {default=info} 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expand map: {default=daemon} 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta from 'daemon' to 'daemon' 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expanded map: {default=daemon} 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expand map: {default=debug} 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta from 'debug' to 'debug' 2024-05-06T12:24:39.108 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: expand_channel_meta expanded map: {default=debug} 2024-05-06T12:24:39.136 INFO:teuthology.orchestra.run.smithi067.stderr:Setting mon public_network to 172.21.0.0/20 2024-05-06T12:24:39.335 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 bash[22393]: f5b02626cc9d13f6a751addaf5eeb80b78c29865215e5d9e4a9cad98b5889c17 2024-05-06T12:24:39.335 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 systemd[1]: Started Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:24:39.335 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T12:24:39.335 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: monmap e1: 1 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} 2024-05-06T12:24:39.335 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: fsmap 2024-05-06T12:24:39.335 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T12:24:39.335 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:39 smithi067 ceph-mon[22413]: mgrmap e1: no daemons active 2024-05-06T12:24:40.076 INFO:teuthology.orchestra.run.smithi067.stderr:Wrote config to /etc/ceph/ceph.conf 2024-05-06T12:24:40.077 INFO:teuthology.orchestra.run.smithi067.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-06T12:24:40.077 INFO:teuthology.orchestra.run.smithi067.stderr:Creating mgr... 2024-05-06T12:24:40.077 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying port 9283 ... 2024-05-06T12:24:40.258 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to reset failed state of unit ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service: Unit ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service not loaded. 2024-05-06T12:24:40.266 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/ceph-847b865c-0ba3-11ef-bc97-c7b262605968.target.wants/ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-847b865c-0ba3-11ef-bc97-c7b262605968@.service. 2024-05-06T12:24:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:40 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1238389322' entity='client.admin' 2024-05-06T12:24:40.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:40 smithi067 podman[22630]: 2024-05-06 12:24:40.656877639 +0000 UTC m=+0.124071557 container create 156e546a6ad85f5765091e8f9e19132fce45286bbfe20f1dd89f4f1f5ff0c6d7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD) 2024-05-06T12:24:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-05-06T12:24:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-05-06T12:24:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-05-06T12:24:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to enable service . firewalld.service is not available 2024-05-06T12:24:41.015 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-05-06T12:24:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-05-06T12:24:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-05-06T12:24:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-05-06T12:24:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr to start... 2024-05-06T12:24:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr... 2024-05-06T12:24:41.037 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:40 smithi067 podman[22630]: 2024-05-06 12:24:40.884926243 +0000 UTC m=+0.352120132 container init 156e546a6ad85f5765091e8f9e19132fce45286bbfe20f1dd89f4f1f5ff0c6d7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vcs-type=git, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, ceph=True, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_BRANCH=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-06T12:24:41.037 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:40 smithi067 podman[22630]: 2024-05-06 12:24:40.887961651 +0000 UTC m=+0.355155525 container start 156e546a6ad85f5765091e8f9e19132fce45286bbfe20f1dd89f4f1f5ff0c6d7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8) 2024-05-06T12:24:41.037 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:40 smithi067 bash[22630]: 156e546a6ad85f5765091e8f9e19132fce45286bbfe20f1dd89f4f1f5ff0c6d7 2024-05-06T12:24:41.037 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:40 smithi067 systemd[1]: Started Ceph mgr.y for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:24:41.355 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:41 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:41.034+0000 7fa814206000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:24:41.356 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:41 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:41.176+0000 7fa814206000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:24:41.356 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:41 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:41.279+0000 7fa814206000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:24:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-05-06T12:24:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsid": "847b865c-0ba3-11ef-bc97-c7b262605968", 2024-05-06T12:24:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "health": { 2024-05-06T12:24:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-06T12:24:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "checks": {}, 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mutes": [] 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum": [ 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 0 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "a" 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "monmap": { 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osdmap": { 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgmap": { 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-06T12:24:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsmap": { 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": false, 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modules": [ 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "iostat", 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "nfs", 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "restful" 2024-05-06T12:24:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "servicemap": { 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modified": "2024-05-06T12:24:34.566968+0000", 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-06T12:24:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-05-06T12:24:41.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2753806830' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T12:24:41.900 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:41 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:41.614+0000 7fa814206000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:24:41.900 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:41 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:41.700+0000 7fa814206000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:24:42.155 INFO:teuthology.orchestra.run.smithi067.stderr:mgr not available, waiting (1/15)... 2024-05-06T12:24:43.165 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:42.903+0000 7fa814206000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:24:43.166 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:43.079+0000 7fa814206000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:24:43.166 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:43.164+0000 7fa814206000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:24:43.768 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:43.488+0000 7fa814206000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:24:43.768 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:43.582+0000 7fa814206000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:24:43.768 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:43.678+0000 7fa814206000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:24:44.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:43.926+0000 7fa814206000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:24:44.663 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:44.397+0000 7fa814206000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:24:44.719 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:44.719 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-05-06T12:24:44.719 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsid": "847b865c-0ba3-11ef-bc97-c7b262605968", 2024-05-06T12:24:44.719 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "health": { 2024-05-06T12:24:44.719 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-06T12:24:44.719 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "checks": {}, 2024-05-06T12:24:44.719 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mutes": [] 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum": [ 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 0 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "a" 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "monmap": { 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-06T12:24:44.720 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osdmap": { 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-06T12:24:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgmap": { 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsmap": { 2024-05-06T12:24:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": false, 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modules": [ 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "iostat", 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "nfs", 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "restful" 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:44.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "servicemap": { 2024-05-06T12:24:44.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:44.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modified": "2024-05-06T12:24:34.566968+0000", 2024-05-06T12:24:44.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-05-06T12:24:44.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:44.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-06T12:24:44.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-05-06T12:24:45.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:45 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1759749384' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T12:24:45.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:44.917+0000 7fa814206000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:24:45.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:45.013+0000 7fa814206000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:24:45.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:45.320+0000 7fa814206000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:24:45.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:45.412+0000 7fa814206000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:24:46.084 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:45.827+0000 7fa814206000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:24:46.084 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:45.963+0000 7fa814206000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:24:46.348 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:46.059+0000 7fa814206000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:24:46.349 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:46.177+0000 7fa814206000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:24:46.349 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:46.268+0000 7fa814206000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:24:47.217 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:46.841+0000 7fa814206000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:24:47.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:47.215+0000 7fa814206000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:24:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: Activating manager daemon y 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: mgrmap e2: y(active, starting, since 0.00269543s) 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: Manager daemon y is now available 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' 2024-05-06T12:24:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:47 smithi067 ceph-mon[22413]: from='mgr.14100 172.21.15.67:0/1084330084' entity='mgr.y' 2024-05-06T12:24:49.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:49 smithi067 ceph-mon[22413]: mgrmap e3: y(active, since 1.00643s) 2024-05-06T12:24:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:mgr not available, waiting (2/15)... 2024-05-06T12:24:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:51 smithi067 ceph-mon[22413]: mgrmap e4: y(active, since 2s) 2024-05-06T12:24:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-05-06T12:24:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsid": "847b865c-0ba3-11ef-bc97-c7b262605968", 2024-05-06T12:24:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "health": { 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "checks": {}, 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mutes": [] 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum": [ 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 0 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "a" 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_age": 14, 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "monmap": { 2024-05-06T12:24:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osdmap": { 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-06T12:24:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-06T12:24:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-06T12:24:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgmap": { 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsmap": { 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-06T12:24:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": true, 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modules": [ 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "iostat", 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "nfs", 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "restful" 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "servicemap": { 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modified": "2024-05-06T12:24:34.566968+0000", 2024-05-06T12:24:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-05-06T12:24:53.734 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-05-06T12:24:53.734 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-06T12:24:53.734 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-05-06T12:24:53.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:53 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/4031898479' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T12:24:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:mgr is available 2024-05-06T12:24:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [global] 2024-05-06T12:24:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: fsid = 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:24:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-06T12:24:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [mgr] 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [osd] 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-06T12:24:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-06T12:24:55.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:55 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1509834701' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T12:24:55.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:55 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1509834701' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-06T12:24:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling cephadm module... 2024-05-06T12:24:56.584 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:56 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1919363674' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-06T12:24:56.585 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setuser ceph since I am not root 2024-05-06T12:24:56.585 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setgroup ceph since I am not root 2024-05-06T12:24:56.585 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:56.447+0000 7f8a126c1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:24:56.585 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:56.582+0000 7f8a126c1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:24:56.870 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:56.685+0000 7f8a126c1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:24:57.148 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:57 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:57.006+0000 7f8a126c1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:24:57.148 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:57 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:57.092+0000 7f8a126c1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:24:57.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:57 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1919363674' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-06T12:24:57.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:57 smithi067 ceph-mon[22413]: mgrmap e5: y(active, since 9s) 2024-05-06T12:24:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-05-06T12:24:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 5, 2024-05-06T12:24:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": true, 2024-05-06T12:24:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-06T12:24:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-06T12:24:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-05-06T12:24:58.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:24:58 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/4284667380' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T12:24:58.543 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:58 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:58.285+0000 7f8a126c1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:24:58.543 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:58 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:58.456+0000 7f8a126c1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:24:58.543 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:58 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:58.541+0000 7f8a126c1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:24:58.655 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for the mgr to restart... 2024-05-06T12:24:58.655 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr epoch 5... 2024-05-06T12:24:59.209 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:58 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:58.876+0000 7f8a126c1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:24:59.209 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:58 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:58.973+0000 7f8a126c1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:24:59.210 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:59.072+0000 7f8a126c1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:24:59.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:59.324+0000 7f8a126c1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:25:00.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:24:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:24:59.783+0000 7f8a126c1000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:25:00.701 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:00.298+0000 7f8a126c1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:25:00.701 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:00.394+0000 7f8a126c1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:25:00.701 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:00.699+0000 7f8a126c1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:25:00.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:00.791+0000 7f8a126c1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:25:01.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:01 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:01.208+0000 7f8a126c1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:25:01.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:01 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:01.344+0000 7f8a126c1000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:25:01.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:01 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:01.440+0000 7f8a126c1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:25:01.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:01 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:01.564+0000 7f8a126c1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:25:01.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:01 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:01.656+0000 7f8a126c1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:25:02.597 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:02 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:02.225+0000 7f8a126c1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:25:02.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:02 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:02.595+0000 7f8a126c1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:25:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-05-06T12:25:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-05-06T12:25:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "initialized": true 2024-05-06T12:25:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: Active manager daemon y restarted 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: Activating manager daemon y 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: osdmap e2: 0 total, 0 up, 0 in 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: mgrmap e6: y(active, starting, since 0.0524679s) 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:25:03.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: Manager daemon y is now available 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:25:03.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:03 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:25:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:mgr epoch 5 is available 2024-05-06T12:25:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:Setting orchestrator backend to cephadm... 2024-05-06T12:25:04.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:04 smithi067 ceph-mon[22413]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-06T12:25:04.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:04 smithi067 ceph-mon[22413]: mgrmap e7: y(active, since 1.05512s) 2024-05-06T12:25:04.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:04 smithi067 ceph-mon[22413]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-06T12:25:04.899 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:25:04] ENGINE Bus STARTING 2024-05-06T12:25:04.899 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:25:04] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:25:04.899 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:25:04] ENGINE Bus STARTED 2024-05-06T12:25:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: [06/May/2024:12:25:04] ENGINE Bus STARTING 2024-05-06T12:25:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: [06/May/2024:12:25:04] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:25:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: [06/May/2024:12:25:04] ENGINE Bus STARTED 2024-05-06T12:25:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:05 smithi067 ceph-mon[22413]: mgrmap e8: y(active, since 2s) 2024-05-06T12:25:06.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: value unchanged 2024-05-06T12:25:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:Generating ssh key... 2024-05-06T12:25:06.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:06 smithi067 ceph-mon[22413]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:07 smithi067 ceph-mon[22413]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:07 smithi067 ceph-mon[22413]: Generating ssh key... 2024-05-06T12:25:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:07 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:07 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: Generating public/private rsa key pair. 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: Your identification has been saved in /tmp/tmpijrs937h/key. 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: Your public key has been saved in /tmp/tmpijrs937h/key.pub. 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: The key fingerprint is: 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: SHA256:qyaArE/bhMbjEcMw8hJ5s0YA4cNCpLKDqek7Do9yCdU ceph-847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: The key's randomart image is: 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: +---[RSA 3072]----+ 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |*+ | 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |=o | 2024-05-06T12:25:07.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |X++. | 2024-05-06T12:25:07.727 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |=@ooE | 2024-05-06T12:25:07.727 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |BoB S | 2024-05-06T12:25:07.727 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |+B.+ . | 2024-05-06T12:25:07.727 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |=.Bo. . | 2024-05-06T12:25:07.727 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |=Bo*. .. | 2024-05-06T12:25:07.727 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: |+*B .o. | 2024-05-06T12:25:07.727 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: +----[SHA256]-----+ 2024-05-06T12:25:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpCj1nTdpQxTR9Ixwh9C2avvgAlAuJOcUx8irByuLJxqT9afDEGLPdOpw7VvabBEPoryTP45evM5/cFmXdneI0XkGIphQsjb6Z0ibBImeMoxJJXh82gyOVvViyzmBkmVfNMjposZB6L7s+23fM9DlAmVsMkCFsACqQebe6sWyUcg2NZ75t2wWyfXzS9TYFfxIhASumASv3ILcHCxg+CcytkIknCbEYJ9q429syZHD8y+95o2kvC2Y2MC+sfp1U2HIUDhAkDHhZMvyAkxX0g3JWBYHumuq4FAgdqpicoCGULsLB16A5lZ/vKupJSW96QP6p9tBwSrgvxBEV21Y6yIkAdZe1Xj0nvKfxODljI+8ZPVcNCv8dZJeTLeVJVNO56/tjE1TaV5NUMSvCQgl+EejQ64JUxWfxrEJO1cHWlmbLOKZTWu+8/g3caAa1WDVkVVA/d04eWmTwu+6sDEINvXWTb/5MeV+kjkC0GInj+gsx0zMzBP4G6os9WIK1oQLQ/vs= ceph-847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:25:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-06T12:25:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:Adding key to root@localhost authorized_keys... 2024-05-06T12:25:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:Adding host smithi067... 2024-05-06T12:25:10.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:10 smithi067 ceph-mon[22413]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:10.771 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Added host 'smithi067' with addr '172.21.15.67' 2024-05-06T12:25:11.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:11 smithi067 ceph-mon[22413]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:11.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:11 smithi067 ceph-mon[22413]: Deploying cephadm binary to smithi067 2024-05-06T12:25:11.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:11 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:11.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:11 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying unmanaged mon service... 2024-05-06T12:25:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled mon update... 2024-05-06T12:25:12.295 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:12 smithi067 ceph-mon[22413]: Added host smithi067 2024-05-06T12:25:12.295 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:12 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:12.719 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying unmanaged mgr service... 2024-05-06T12:25:13.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:13 smithi067 ceph-mon[22413]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:13.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:13 smithi067 ceph-mon[22413]: Saving service mon spec with placement count:5 2024-05-06T12:25:13.505 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-05-06T12:25:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:14 smithi067 ceph-mon[22413]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:14 smithi067 ceph-mon[22413]: Saving service mgr spec with placement count:2 2024-05-06T12:25:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:14 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:15.852 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:15 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:15.852 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:15 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:16.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:16 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/865461947' entity='client.admin' 2024-05-06T12:25:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling the dashboard module... 2024-05-06T12:25:18.587 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:18 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1728641695' entity='client.admin' 2024-05-06T12:25:18.587 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:18 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:19.903 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:19 smithi067 ceph-mon[22413]: from='mgr.14118 172.21.15.67:0/3915057084' entity='mgr.y' 2024-05-06T12:25:19.903 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:19 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3380695933' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-06T12:25:19.903 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:19 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setuser ceph since I am not root 2024-05-06T12:25:19.903 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:19 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setgroup ceph since I am not root 2024-05-06T12:25:19.903 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:19 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:19.764+0000 7f82dcb44000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:25:19.903 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:19 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:19.898+0000 7f82dcb44000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:25:20.211 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:20.001+0000 7f82dcb44000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:25:20.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:20.319+0000 7f82dcb44000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:25:20.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:20.405+0000 7f82dcb44000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:25:20.865 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:20 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3380695933' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-06T12:25:20.866 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:20 smithi067 ceph-mon[22413]: mgrmap e9: y(active, since 16s) 2024-05-06T12:25:21.207 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-05-06T12:25:21.207 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 9, 2024-05-06T12:25:21.207 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": true, 2024-05-06T12:25:21.207 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-06T12:25:21.207 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-06T12:25:21.207 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-05-06T12:25:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:21 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1149340402' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T12:25:21.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:21 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:21.611+0000 7f82dcb44000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:25:21.841 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for the mgr to restart... 2024-05-06T12:25:21.842 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr epoch 9... 2024-05-06T12:25:22.090 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:21 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:21.787+0000 7f82dcb44000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:25:22.090 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:21 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:21.878+0000 7f82dcb44000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:25:22.409 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:22.212+0000 7f82dcb44000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:25:22.409 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:22.310+0000 7f82dcb44000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:25:22.409 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:22.407+0000 7f82dcb44000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:25:22.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:22.657+0000 7f82dcb44000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:25:23.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:23.115+0000 7f82dcb44000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:25:23.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:23.617+0000 7f82dcb44000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:25:23.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:23.713+0000 7f82dcb44000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:25:24.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:24.019+0000 7f82dcb44000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:25:24.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:24.111+0000 7f82dcb44000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:25:24.880 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:24.528+0000 7f82dcb44000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:25:24.880 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:24.665+0000 7f82dcb44000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:25:24.880 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:24.761+0000 7f82dcb44000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:25:24.880 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:24.878+0000 7f82dcb44000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:25:25.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:24.970+0000 7f82dcb44000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:25:25.912 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:25.537+0000 7f82dcb44000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:25:26.162 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:25:25.910+0000 7f82dcb44000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:25:26.162 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:25 smithi067 ceph-mon[22413]: Active manager daemon y restarted 2024-05-06T12:25:26.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:25 smithi067 ceph-mon[22413]: Activating manager daemon y 2024-05-06T12:25:26.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:25 smithi067 ceph-mon[22413]: osdmap e3: 0 total, 0 up, 0 in 2024-05-06T12:25:26.954 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:26 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:25:26] ENGINE Bus STARTING 2024-05-06T12:25:26.955 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:26 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:25:26] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:25:26.955 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:25:26 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:25:26] ENGINE Bus STARTED 2024-05-06T12:25:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-05-06T12:25:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-05-06T12:25:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "initialized": true 2024-05-06T12:25:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-05-06T12:25:27.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: mgrmap e10: y(active, starting, since 0.0551625s) 2024-05-06T12:25:27.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: Manager daemon y is now available 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:25:27.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:27.848 INFO:teuthology.orchestra.run.smithi067.stderr:mgr epoch 9 is available 2024-05-06T12:25:27.848 INFO:teuthology.orchestra.run.smithi067.stderr:Generating a dashboard self-signed certificate... 2024-05-06T12:25:28.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:27 smithi067 ceph-mon[22413]: [06/May/2024:12:25:26] ENGINE Bus STARTING 2024-05-06T12:25:28.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:27 smithi067 ceph-mon[22413]: [06/May/2024:12:25:26] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:25:28.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:27 smithi067 ceph-mon[22413]: [06/May/2024:12:25:26] ENGINE Bus STARTED 2024-05-06T12:25:28.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:27 smithi067 ceph-mon[22413]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-06T12:25:28.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:27 smithi067 ceph-mon[22413]: mgrmap e11: y(active, since 1.06509s) 2024-05-06T12:25:28.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:27 smithi067 ceph-mon[22413]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-06T12:25:28.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:27 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:28.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Self-signed certificate created 2024-05-06T12:25:29.204 INFO:teuthology.orchestra.run.smithi067.stderr:Creating initial admin user... 2024-05-06T12:25:29.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:29 smithi067 ceph-mon[22413]: mgrmap e12: y(active, since 2s) 2024-05-06T12:25:29.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:29 smithi067 ceph-mon[22413]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:29.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:29 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:29.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:29 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:29.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:29 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$hFnEDkqdpzqxDV8qYBX/.e7Np4Z1NcM4MSWLHQ07S3BbwXRLIuOJy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714998330, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-06T12:25:30.976 INFO:teuthology.orchestra.run.smithi067.stderr:Fetching dashboard port number... 2024-05-06T12:25:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:31 smithi067 ceph-mon[22413]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:31 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:32.047 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 8443 2024-05-06T12:25:32.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:32 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/4264608161' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-06T12:25:32.654 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-05-06T12:25:32.661 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-05-06T12:25:32.662 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-05-06T12:25:32.662 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-06T12:25:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:Ceph Dashboard is now available at: 2024-05-06T12:25:32.664 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:32.664 INFO:teuthology.orchestra.run.smithi067.stderr: URL: https://smithi067:8443/ 2024-05-06T12:25:32.664 INFO:teuthology.orchestra.run.smithi067.stderr: User: admin 2024-05-06T12:25:32.664 INFO:teuthology.orchestra.run.smithi067.stderr: Password: dm8587sn91 2024-05-06T12:25:32.664 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling autotune for osd_memory_target 2024-05-06T12:25:33.187 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:33 smithi067 ceph-mon[22413]: mgrmap e13: y(active, since 6s) 2024-05-06T12:25:34.702 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:You can access the Ceph CLI with: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:Please consider enabling telemetry to help improve Ceph: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: ceph telemetry on 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:For more information see: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-05-06T12:25:35.371 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:25:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:Bootstrap complete. 2024-05-06T12:25:35.412 INFO:tasks.cephadm:Fetching config... 2024-05-06T12:25:35.413 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:25:35.413 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-06T12:25:35.431 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-06T12:25:35.431 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:25:35.431 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-06T12:25:35.486 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-06T12:25:35.486 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:25:35.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-06T12:25:35.550 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-06T12:25:35.550 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:25:35.550 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-06T12:25:35.605 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-06T12:25:35.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpCj1nTdpQxTR9Ixwh9C2avvgAlAuJOcUx8irByuLJxqT9afDEGLPdOpw7VvabBEPoryTP45evM5/cFmXdneI0XkGIphQsjb6Z0ibBImeMoxJJXh82gyOVvViyzmBkmVfNMjposZB6L7s+23fM9DlAmVsMkCFsACqQebe6sWyUcg2NZ75t2wWyfXzS9TYFfxIhASumASv3ILcHCxg+CcytkIknCbEYJ9q429syZHD8y+95o2kvC2Y2MC+sfp1U2HIUDhAkDHhZMvyAkxX0g3JWBYHumuq4FAgdqpicoCGULsLB16A5lZ/vKupJSW96QP6p9tBwSrgvxBEV21Y6yIkAdZe1Xj0nvKfxODljI+8ZPVcNCv8dZJeTLeVJVNO56/tjE1TaV5NUMSvCQgl+EejQ64JUxWfxrEJO1cHWlmbLOKZTWu+8/g3caAa1WDVkVVA/d04eWmTwu+6sDEINvXWTb/5MeV+kjkC0GInj+gsx0zMzBP4G6os9WIK1oQLQ/vs= ceph-847b865c-0ba3-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-06T12:25:35.681 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpCj1nTdpQxTR9Ixwh9C2avvgAlAuJOcUx8irByuLJxqT9afDEGLPdOpw7VvabBEPoryTP45evM5/cFmXdneI0XkGIphQsjb6Z0ibBImeMoxJJXh82gyOVvViyzmBkmVfNMjposZB6L7s+23fM9DlAmVsMkCFsACqQebe6sWyUcg2NZ75t2wWyfXzS9TYFfxIhASumASv3ILcHCxg+CcytkIknCbEYJ9q429syZHD8y+95o2kvC2Y2MC+sfp1U2HIUDhAkDHhZMvyAkxX0g3JWBYHumuq4FAgdqpicoCGULsLB16A5lZ/vKupJSW96QP6p9tBwSrgvxBEV21Y6yIkAdZe1Xj0nvKfxODljI+8ZPVcNCv8dZJeTLeVJVNO56/tjE1TaV5NUMSvCQgl+EejQ64JUxWfxrEJO1cHWlmbLOKZTWu+8/g3caAa1WDVkVVA/d04eWmTwu+6sDEINvXWTb/5MeV+kjkC0GInj+gsx0zMzBP4G6os9WIK1oQLQ/vs= ceph-847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:25:35.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpCj1nTdpQxTR9Ixwh9C2avvgAlAuJOcUx8irByuLJxqT9afDEGLPdOpw7VvabBEPoryTP45evM5/cFmXdneI0XkGIphQsjb6Z0ibBImeMoxJJXh82gyOVvViyzmBkmVfNMjposZB6L7s+23fM9DlAmVsMkCFsACqQebe6sWyUcg2NZ75t2wWyfXzS9TYFfxIhASumASv3ILcHCxg+CcytkIknCbEYJ9q429syZHD8y+95o2kvC2Y2MC+sfp1U2HIUDhAkDHhZMvyAkxX0g3JWBYHumuq4FAgdqpicoCGULsLB16A5lZ/vKupJSW96QP6p9tBwSrgvxBEV21Y6yIkAdZe1Xj0nvKfxODljI+8ZPVcNCv8dZJeTLeVJVNO56/tjE1TaV5NUMSvCQgl+EejQ64JUxWfxrEJO1cHWlmbLOKZTWu+8/g3caAa1WDVkVVA/d04eWmTwu+6sDEINvXWTb/5MeV+kjkC0GInj+gsx0zMzBP4G6os9WIK1oQLQ/vs= ceph-847b865c-0ba3-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-06T12:25:35.732 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpCj1nTdpQxTR9Ixwh9C2avvgAlAuJOcUx8irByuLJxqT9afDEGLPdOpw7VvabBEPoryTP45evM5/cFmXdneI0XkGIphQsjb6Z0ibBImeMoxJJXh82gyOVvViyzmBkmVfNMjposZB6L7s+23fM9DlAmVsMkCFsACqQebe6sWyUcg2NZ75t2wWyfXzS9TYFfxIhASumASv3ILcHCxg+CcytkIknCbEYJ9q429syZHD8y+95o2kvC2Y2MC+sfp1U2HIUDhAkDHhZMvyAkxX0g3JWBYHumuq4FAgdqpicoCGULsLB16A5lZ/vKupJSW96QP6p9tBwSrgvxBEV21Y6yIkAdZe1Xj0nvKfxODljI+8ZPVcNCv8dZJeTLeVJVNO56/tjE1TaV5NUMSvCQgl+EejQ64JUxWfxrEJO1cHWlmbLOKZTWu+8/g3caAa1WDVkVVA/d04eWmTwu+6sDEINvXWTb/5MeV+kjkC0GInj+gsx0zMzBP4G6os9WIK1oQLQ/vs= ceph-847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:25:35.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-06T12:25:35.905 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:35 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/4203147047' entity='client.admin' 2024-05-06T12:25:38.673 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:38.673 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:25:38.673 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:38.673 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:38.673 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:39.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:39 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/496240331' entity='client.admin' 2024-05-06T12:25:39.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:39.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:40.117 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-06T12:25:40.118 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-06T12:25:42.793 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-05-06T12:25:42.793 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-06T12:25:42.793 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-05-06T12:25:42.812 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-06T12:25:42.812 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:25:42.870 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-05-06T12:25:42.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch host add smithi086 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:45.657 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi086' with addr '172.21.15.86' 2024-05-06T12:25:45.963 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:45 smithi067 ceph-mon[22413]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:45.964 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:45 smithi067 ceph-mon[22413]: Deploying cephadm binary to smithi086 2024-05-06T12:25:45.964 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:45 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:45.964 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:45 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:45.964 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:45 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:25:45.964 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:45 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:25:46.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-06T12:25:47.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:46 smithi067 ceph-mon[22413]: Added host smithi086 2024-05-06T12:25:47.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:46 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:47.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:46 smithi067 ceph-mon[22413]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:47.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:46 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:48.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:25:48.142 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2024-05-06T12:25:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:48 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:48.770 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-06T12:25:48.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-06T12:25:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:49 smithi067 ceph-mon[22413]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:49 smithi067 ceph-mon[22413]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T12:25:51.489 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-05-06T12:25:51.626 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:51 smithi067 ceph-mon[22413]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:51.626 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:51 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:51.626 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:51 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:51.627 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:51 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:25:51.627 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:51 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:51.627 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:51 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3993225408' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-06T12:25:52.220 INFO:tasks.cephadm:Adding mon.a on smithi067 2024-05-06T12:25:52.220 INFO:tasks.cephadm:Adding mon.c on smithi067 2024-05-06T12:25:52.220 INFO:tasks.cephadm:Adding mon.b on smithi086 2024-05-06T12:25:52.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch apply mon '3;smithi067:172.21.15.67=a;smithi067:[v2:172.21.15.67:3301,v1:172.21.15.67:6790]=c;smithi086:172.21.15.86=b' 2024-05-06T12:25:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:52 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3993225408' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-06T12:25:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:52 smithi067 ceph-mon[22413]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T12:25:53.656 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:53 smithi067 ceph-mon[22413]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:54.055 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2024-05-06T12:25:54.736 DEBUG:teuthology.orchestra.run.smithi067:mon.c> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.c.service 2024-05-06T12:25:54.739 DEBUG:teuthology.orchestra.run.smithi086:mon.b> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.b.service 2024-05-06T12:25:54.741 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-06T12:25:54.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T12:25:55.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi067:[v2:172.21.15.67:3301,v1:172.21.15.67:6790]=c;smithi086:172.21.15.86=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi067:[v2:172.21.15.67:3301,v1:172.21.15.67:6790]=c;smithi086:172.21.15.86=b;count:3 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:55.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:25:55.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:25:55.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:55 smithi067 ceph-mon[22413]: Deploying daemon mon.c on smithi067 2024-05-06T12:25:56.636 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:25:56.636 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"847b865c-0ba3-11ef-bc97-c7b262605968","modified":"2024-05-06T12:24:31.741125Z","created":"2024-05-06T12:24:31.741125Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T12:25:56.638 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-05-06T12:25:57.362 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[22413]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:57.362 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/1397915078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T12:25:57.362 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 systemd[1]: Starting Ceph mon.c for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 podman[27118]: 2024-05-06 12:25:57.264098495 +0000 UTC m=+0.018957421 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 podman[27118]: 2024-05-06 12:25:57.368329715 +0000 UTC m=+0.123188622 container create d0561ab1fc3ce617bcc6b9ac11f061c8dcf3e52720a51ef078259a2b4c20fa0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-type=git, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, release=754) 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 podman[27118]: 2024-05-06 12:25:57.570906653 +0000 UTC m=+0.325765561 container init d0561ab1fc3ce617bcc6b9ac11f061c8dcf3e52720a51ef078259a2b4c20fa0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, ceph=True, vcs-type=git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0) 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 podman[27118]: 2024-05-06 12:25:57.574440358 +0000 UTC m=+0.329299261 container start d0561ab1fc3ce617bcc6b9ac11f061c8dcf3e52720a51ef078259a2b4c20fa0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vendor=Red Hat, Inc.) 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: pidfile_write: ignore empty --pid-file 2024-05-06T12:25:57.617 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: load: jerasure load: lrc 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: RocksDB version: 6.15.5 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Compile date Apr 18 2022 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: DB SUMMARY 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: DB Session ID: R2HFDEVC5DJ273UGHI9R 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: CURRENT file: CURRENT 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.error_if_exists: 0 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.create_if_missing: 0 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.paranoid_checks: 1 2024-05-06T12:25:57.618 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T12:25:57.619 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.env: 0x55d437ec6860 2024-05-06T12:25:57.619 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.fs: Posix File System 2024-05-06T12:25:57.619 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.info_log: 0x55d43a2c7dc0 2024-05-06T12:25:57.619 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T12:25:57.619 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.statistics: (nil) 2024-05-06T12:25:57.619 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.use_fsync: 0 2024-05-06T12:25:57.620 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_log_file_size: 0 2024-05-06T12:25:57.620 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T12:25:57.620 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T12:25:57.620 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.allow_fallocate: 1 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.use_direct_reads: 0 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.db_log_dir: 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T12:25:57.621 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.write_buffer_manager: 0x55d43a3b82d0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T12:25:57.622 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.unordered_write: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.row_cache: None 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.wal_filter: None 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.preserve_deletes: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.two_write_queues: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.atomic_flush: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T12:25:57.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.log_readahead_size: 0 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_background_jobs: 2 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_background_compactions: -1 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_subcompactions: 1 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T12:25:57.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_open_files: -1 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_background_flushes: -1 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Compression algorithms supported: 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kZSTD supported: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kXpressCompression supported: 0 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T12:25:57.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kLZ4Compression supported: 1 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kBZip2Compression supported: 0 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kZlibCompression supported: 1 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: kSnappyCompression supported: 1 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.merge_operator: 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_filter: None 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T12:25:57.626 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d43a293d00) 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: index_type: 0 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: data_block_index_type: 0 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: index_shortening: 1 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: hash_index_allow_collision: 1 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: checksum: 1 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: no_block_cache: 0 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache: 0x55d43a2fe170 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache_options: 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: capacity : 536870912 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: num_shard_bits : 4 2024-05-06T12:25:57.627 INFO:journalctl@ceph.mon.c.smithi067.stdout: strict_capacity_limit : 0 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache_compressed: (nil) 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: persistent_cache: (nil) 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_size: 4096 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_size_deviation: 10 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_restart_interval: 16 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: index_block_restart_interval: 1 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: metadata_block_size: 4096 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: partition_filters: 0 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: use_delta_encoding: 1 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: whole_key_filtering: 1 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: verify_compression: 0 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-05-06T12:25:57.628 INFO:journalctl@ceph.mon.c.smithi067.stdout: format_version: 4 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout: enable_index_compression: 1 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_align: 0 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression: NoCompression 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.num_levels: 7 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T12:25:57.629 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T12:25:57.630 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.arena_block_size: 4194304 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T12:25:57.631 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.table_properties_collectors: 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.inplace_update_support: 0 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T12:25:57.632 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.bloom_locality: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.max_successive_merges: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.ttl: 2592000 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.enable_blob_files: false 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.min_blob_size: 0 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T12:25:57.633 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714998357601244, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714998357601589, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1714998357, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "0fce8468-99a5-480b-a826-ca7541cfc140", "db_session_id": "R2HFDEVC5DJ273UGHI9R"}} 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714998357601927, "job": 1, "event": "recovery_finished"} 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55d43a2e4a80 2024-05-06T12:25:57.634 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: DB pointer 0x55d43a2f4000 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** DB Stats ** 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:25:57.635 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.7 0.00 0.00 1 0.000 0 0 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T12:25:57.636 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 2024-05-06T12:25:57.637 INFO:journalctl@ceph.mon.c.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.7 0.00 0.00 1 0.000 0 0 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T12:25:57.638 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: using public_addrv [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: starting mon.c rank -1 at public addrs [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] at bind addrs [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(???) e0 preinit fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).mds e1 new map 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout: e1 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout: legacy client fscid: -1 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout: No filesystems configured 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-06T12:25:57.639 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi067:[v2:172.21.15.67:3301,v1:172.21.15.67:6790]=c;smithi086:172.21.15.86=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi067:[v2:172.21.15.67:3301,v1:172.21.15.67:6790]=c;smithi086:172.21.15.86=b;count:3 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:25:57.640 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: Deploying daemon mon.c on smithi067 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/1397915078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-06T12:25:57.641 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 ceph-mon[27137]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-05-06T12:25:57.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 bash[27118]: d0561ab1fc3ce617bcc6b9ac11f061c8dcf3e52720a51ef078259a2b4c20fa0d 2024-05-06T12:25:57.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:25:57 smithi067 systemd[1]: Started Ceph mon.c for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:25:58.141 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-06T12:25:58.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T12:26:02.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: mon.a calling monitor election 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: mon.c calling monitor election 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:02.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: monmap e2: 2 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],c=[v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0]} 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: fsmap 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: mgrmap e13: y(active, since 36s) 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: overall HEALTH_OK 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[22413]: Deploying daemon mon.b on smithi086 2024-05-06T12:26:02.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: mon.a calling monitor election 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: mon.c calling monitor election 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T12:26:02.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: monmap e2: 2 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],c=[v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0]} 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: fsmap 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: mgrmap e13: y(active, since 36s) 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: overall HEALTH_OK 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:02.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:02 smithi067 ceph-mon[27137]: Deploying daemon mon.b on smithi086 2024-05-06T12:26:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:26:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":2,"fsid":"847b865c-0ba3-11ef-bc97-c7b262605968","modified":"2024-05-06T12:25:57.620593Z","created":"2024-05-06T12:24:31.741125Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3301","nonce":0},{"type":"v1","addr":"172.21.15.67:6790","nonce":0}]},"addr":"172.21.15.67:6790/0","public_addr":"172.21.15.67:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-06T12:26:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 2 2024-05-06T12:26:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:03 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/3055491049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T12:26:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:03.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:03 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/3055491049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T12:26:03.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:05 smithi067 ceph-mon[22413]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:05.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:05 smithi067 ceph-mon[27137]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:06.506 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-06T12:26:06.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T12:26:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: mon.a calling monitor election 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: mon.c calling monitor election 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: mon.b calling monitor election 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: monmap e3: 3 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0]} 2024-05-06T12:26:11.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: fsmap 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: mgrmap e13: y(active, since 45s) 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: overall HEALTH_OK 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: mon.a calling monitor election 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: mon.c calling monitor election 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: mon.b calling monitor election 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: monmap e3: 3 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],b=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],c=[v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0]} 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: fsmap 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: mgrmap e13: y(active, since 45s) 2024-05-06T12:26:11.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: overall HEALTH_OK 2024-05-06T12:26:11.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:11.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:11.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:11.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:12.039 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-06T12:26:12.039 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":3,"fsid":"847b865c-0ba3-11ef-bc97-c7b262605968","modified":"2024-05-06T12:26:06.449836Z","created":"2024-05-06T12:24:31.741125Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3301","nonce":0},{"type":"v1","addr":"172.21.15.67:6790","nonce":0}]},"addr":"172.21.15.67:6790/0","public_addr":"172.21.15.67:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-06T12:26:12.041 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 3 2024-05-06T12:26:12.795 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:12 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/2283965881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T12:26:12.795 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:12 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:12.795 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:12 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/2283965881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T12:26:12.795 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:12 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:26:13.026 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-06T12:26:13.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-06T12:26:13.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:13.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:13 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:14.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:14 smithi067 ceph-mon[22413]: Reconfiguring mon.a (unknown last config time)... 2024-05-06T12:26:14.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:14 smithi067 ceph-mon[22413]: Reconfiguring daemon mon.a on smithi067 2024-05-06T12:26:14.652 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:14 smithi067 ceph-mon[27137]: Reconfiguring mon.a (unknown last config time)... 2024-05-06T12:26:14.652 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:14 smithi067 ceph-mon[27137]: Reconfiguring daemon mon.a on smithi067 2024-05-06T12:26:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:14 smithi086 ceph-mon[23164]: Reconfiguring mon.a (unknown last config time)... 2024-05-06T12:26:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:14 smithi086 ceph-mon[23164]: Reconfiguring daemon mon.a on smithi067 2024-05-06T12:26:15.684 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[22413]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[27137]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:15.685 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:15 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:15 smithi086 ceph-mon[23164]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:15 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:15 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:15 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:15 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:15.981 INFO:teuthology.orchestra.run.smithi067.stdout:# minimal ceph.conf for 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:26:15.981 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-05-06T12:26:15.981 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:26:15.981 INFO:teuthology.orchestra.run.smithi067.stdout: mon_host = [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2024-05-06T12:26:16.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[22413]: Reconfiguring mon.c (monmap changed)... 2024-05-06T12:26:16.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[22413]: Reconfiguring daemon mon.c on smithi067 2024-05-06T12:26:16.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2388432795' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:16.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:16.756 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[27137]: Reconfiguring mon.c (monmap changed)... 2024-05-06T12:26:16.757 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[27137]: Reconfiguring daemon mon.c on smithi067 2024-05-06T12:26:16.757 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2388432795' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:16.757 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:16 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:16.781 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-06T12:26:16.781 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:26:16.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-05-06T12:26:16.808 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:26:16.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:26:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:16 smithi086 ceph-mon[23164]: Reconfiguring mon.c (monmap changed)... 2024-05-06T12:26:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:16 smithi086 ceph-mon[23164]: Reconfiguring daemon mon.c on smithi067 2024-05-06T12:26:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:16 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2388432795' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:16 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:16.873 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-06T12:26:16.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-05-06T12:26:16.902 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-06T12:26:16.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:26:16.968 INFO:tasks.cephadm:Adding mgr.y on smithi067 2024-05-06T12:26:16.968 INFO:tasks.cephadm:Adding mgr.x on smithi086 2024-05-06T12:26:16.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch apply mgr '2;smithi067=y;smithi086=x' 2024-05-06T12:26:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:17 smithi086 ceph-mon[23164]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:17 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:17 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:17.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:17 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:17.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:17 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[22413]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:17.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:17.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[27137]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:26:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:26:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:17 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:18.834 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mgr update... 2024-05-06T12:26:18.835 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:18 smithi086 ceph-mon[23164]: Reconfiguring mon.b (monmap changed)... 2024-05-06T12:26:18.835 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:18 smithi086 ceph-mon[23164]: Reconfiguring daemon mon.b on smithi086 2024-05-06T12:26:18.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:18 smithi067 ceph-mon[22413]: Reconfiguring mon.b (monmap changed)... 2024-05-06T12:26:18.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:18 smithi067 ceph-mon[22413]: Reconfiguring daemon mon.b on smithi086 2024-05-06T12:26:18.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:18 smithi067 ceph-mon[27137]: Reconfiguring mon.b (monmap changed)... 2024-05-06T12:26:18.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:18 smithi067 ceph-mon[27137]: Reconfiguring daemon mon.b on smithi086 2024-05-06T12:26:19.365 DEBUG:teuthology.orchestra.run.smithi086:mgr.x> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service 2024-05-06T12:26:19.368 INFO:tasks.cephadm:Deploying OSDs... 2024-05-06T12:26:19.368 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:26:19.368 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T12:26:19.388 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T12:26:19.388 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-05-06T12:26:19.447 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T12:26:19.447 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:19.447 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-06T12:26:19.448 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:19.448 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:19.448 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-06 12:25:57.077406926 +0000 2024-05-06T12:26:19.448 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-06 12:23:40.946572373 +0000 2024-05-06T12:26:19.448 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-06 12:23:40.946572373 +0000 2024-05-06T12:26:19.448 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-06 12:23:40.946572373 +0000 2024-05-06T12:26:19.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T12:26:19.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-06T12:26:19.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-06T12:26:19.515 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00027323 s, 1.9 MB/s 2024-05-06T12:26:19.517 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T12:26:19.577 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-06 12:25:57.077406926 +0000 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-06 12:23:41.295563985 +0000 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-06 12:23:41.295563985 +0000 2024-05-06T12:26:19.635 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-06 12:23:41.295563985 +0000 2024-05-06T12:26:19.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T12:26:19.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-06T12:26:19.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-06T12:26:19.701 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000228333 s, 2.2 MB/s 2024-05-06T12:26:19.702 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T12:26:19.760 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-05-06T12:26:19.817 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T12:26:19.817 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:19.817 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-06T12:26:19.817 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:19.817 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:19.817 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-06 12:25:57.078406903 +0000 2024-05-06T12:26:19.818 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-06 12:23:41.650555452 +0000 2024-05-06T12:26:19.818 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-06 12:23:41.650555452 +0000 2024-05-06T12:26:19.818 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-06 12:23:41.650555452 +0000 2024-05-06T12:26:19.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T12:26:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:19 smithi086 ceph-mon[23164]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:19 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:19.879 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:19 smithi067 ceph-mon[27137]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:19.880 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:19 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:19.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:19 smithi067 ceph-mon[22413]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:19.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:19 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-06T12:26:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-06T12:26:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000244983 s, 2.1 MB/s 2024-05-06T12:26:19.883 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T12:26:19.940 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-05-06T12:26:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-06 12:25:57.078406903 +0000 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-06 12:23:42.008546848 +0000 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-06 12:23:42.008546848 +0000 2024-05-06T12:26:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-06 12:23:42.008546848 +0000 2024-05-06T12:26:19.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T12:26:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-06T12:26:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-06T12:26:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000196955 s, 2.6 MB/s 2024-05-06T12:26:20.061 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T12:26:20.118 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-06T12:26:20.118 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T12:26:20.133 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T12:26:20.133 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-05-06T12:26:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T12:26:20.188 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:20.188 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-06T12:26:20.188 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:20.188 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:20.188 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-06 12:26:05.870490115 +0000 2024-05-06T12:26:20.188 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-06 12:22:43.931854315 +0000 2024-05-06T12:26:20.189 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-06 12:22:43.931854315 +0000 2024-05-06T12:26:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-06 12:22:43.931854315 +0000 2024-05-06T12:26:20.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T12:26:20.250 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-06T12:26:20.250 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-06T12:26:20.250 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000194122 s, 2.6 MB/s 2024-05-06T12:26:20.252 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T12:26:20.308 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-05-06T12:26:20.365 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T12:26:20.365 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:20.365 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-06T12:26:20.365 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:20.365 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:20.365 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-06 12:26:05.871490093 +0000 2024-05-06T12:26:20.366 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-06 12:22:44.268846435 +0000 2024-05-06T12:26:20.366 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-06 12:22:44.268846435 +0000 2024-05-06T12:26:20.366 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-06 12:22:44.268846435 +0000 2024-05-06T12:26:20.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T12:26:20.427 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-06T12:26:20.428 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-06T12:26:20.428 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000204424 s, 2.5 MB/s 2024-05-06T12:26:20.429 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T12:26:20.484 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-05-06T12:26:20.538 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T12:26:20.538 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:20.539 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-06T12:26:20.539 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:20.539 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:20.539 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-06 12:26:05.871490093 +0000 2024-05-06T12:26:20.539 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-06 12:22:44.601838646 +0000 2024-05-06T12:26:20.539 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-06 12:22:44.601838646 +0000 2024-05-06T12:26:20.539 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-06 12:22:44.601838646 +0000 2024-05-06T12:26:20.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T12:26:20.600 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-06T12:26:20.600 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-06T12:26:20.600 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000200884 s, 2.5 MB/s 2024-05-06T12:26:20.601 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T12:26:20.657 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-05-06T12:26:20.711 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-06 12:26:05.871490093 +0000 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-06 12:22:44.954830390 +0000 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-06 12:22:44.954830390 +0000 2024-05-06T12:26:20.712 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-06 12:22:44.954830390 +0000 2024-05-06T12:26:20.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi067=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: Saving service mgr spec with placement smithi067=y;smithi086=x;count:2 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:26:20.713 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:20.774 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-06T12:26:20.775 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-06T12:26:20.775 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000171457 s, 3.0 MB/s 2024-05-06T12:26:20.776 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T12:26:20.833 INFO:tasks.cephadm:Deploying osd.0 on smithi067 with /dev/vg_nvme/lv_4... 2024-05-06T12:26:20.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-06T12:26:20.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi067=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:20.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: Saving service mgr spec with placement smithi067=y;smithi086=x;count:2 2024-05-06T12:26:20.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:20.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi067=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: Saving service mgr spec with placement smithi067=y;smithi086=x;count:2 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:20.861 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:20.862 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:20.862 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:26:20.862 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-06T12:26:20.862 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:26:20.862 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:21 smithi067 ceph-mon[22413]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:21 smithi067 ceph-mon[22413]: Deploying daemon mgr.x on smithi086 2024-05-06T12:26:21.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:21 smithi067 ceph-mon[27137]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:21 smithi067 ceph-mon[27137]: Deploying daemon mgr.x on smithi086 2024-05-06T12:26:21.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:21 smithi086 ceph-mon[23164]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:21.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:21 smithi086 ceph-mon[23164]: Deploying daemon mgr.x on smithi086 2024-05-06T12:26:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-06T12:26:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-06T12:26:22.876 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-06T12:26:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-06T12:26:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112619 s, 931 MB/s 2024-05-06T12:26:22.877 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 2024-05-06T12:26:22.877 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-06T12:26:23.093 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:22.783+0000 7f755a922000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:26:23.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:23.107+0000 7f755a922000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:26:23.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:23.192+0000 7f755a922000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:26:23.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-05-06T12:26:23.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[27137]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:23.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:23.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:23.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[22413]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:23 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:23.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:23 smithi086 ceph-mon[23164]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:23.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:23 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:23 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:23 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:23 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:24.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:24.327+0000 7f755a922000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:24.488+0000 7f755a922000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:24.572+0000 7f755a922000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:24 smithi086 ceph-mon[23164]: Reconfiguring mgr.y (unknown last config time)... 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:24.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:24 smithi086 ceph-mon[23164]: Reconfiguring daemon mgr.y on smithi067 2024-05-06T12:26:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[22413]: Reconfiguring mgr.y (unknown last config time)... 2024-05-06T12:26:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:26:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[22413]: Reconfiguring daemon mgr.y on smithi067 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[27137]: Reconfiguring mgr.y (unknown last config time)... 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:24 smithi067 ceph-mon[27137]: Reconfiguring daemon mgr.y on smithi067 2024-05-06T12:26:25.294 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:24.879+0000 7f755a922000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:26:25.294 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:24.966+0000 7f755a922000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:26:25.294 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:25.056+0000 7f755a922000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:26:25.294 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:25.292+0000 7f755a922000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:26:25.760 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:25 smithi067 ceph-mon[22413]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:25.760 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:25 smithi067 ceph-mon[27137]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:25 smithi086 ceph-mon[23164]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:25.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:25.726+0000 7f755a922000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:26:26.560 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:26.223+0000 7f755a922000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:26:26.560 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:26.317+0000 7f755a922000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:26:26.657 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:26:26.657 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:26:26.657 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:26.657 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:26:26.657 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:26:26.658 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:26.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:26.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:26.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:26.606+0000 7f755a922000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:26:26.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:26.693+0000 7f755a922000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:27.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:27.089+0000 7f755a922000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:26:27.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:27.219+0000 7f755a922000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:26:27.370 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:27.309+0000 7f755a922000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:26:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:27 smithi067 ceph-mon[22413]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:27 smithi067 ceph-mon[22413]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:27.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:27 smithi067 ceph-mon[27137]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:27.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:27 smithi067 ceph-mon[27137]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:27.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:27 smithi086 ceph-mon[23164]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:27.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:27 smithi086 ceph-mon[23164]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:27.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:27.421+0000 7f755a922000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:26:27.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:27.508+0000 7f755a922000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:26:28.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:28.047+0000 7f755a922000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:26:28.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:26:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:26:28.420+0000 7f755a922000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:26:28.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2748116469' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]: dispatch 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]: dispatch 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]': finished 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: osdmap e5: 1 total, 0 up, 1 in 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: Standby manager daemon x started 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:26:28.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:28 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:26:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2748116469' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]: dispatch 2024-05-06T12:26:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]: dispatch 2024-05-06T12:26:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]': finished 2024-05-06T12:26:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: osdmap e5: 1 total, 0 up, 1 in 2024-05-06T12:26:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: Standby manager daemon x started 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2748116469' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8669559-459e-464d-a02a-bc107141d715"}]': finished 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: osdmap e5: 1 total, 0 up, 1 in 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: Standby manager daemon x started 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:26:28.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:26:28.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:26:28.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:28 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/408404544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:26:29.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:29 smithi086 ceph-mon[23164]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:29.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:29 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1575277568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:26:29.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:29 smithi086 ceph-mon[23164]: mgrmap e14: y(active, since 63s), standbys: x 2024-05-06T12:26:29.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:29 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:26:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[22413]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1575277568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:26:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[22413]: mgrmap e14: y(active, since 63s), standbys: x 2024-05-06T12:26:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:26:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[27137]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1575277568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:26:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[27137]: mgrmap e14: y(active, since 63s), standbys: x 2024-05-06T12:26:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:29 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:26:31.602 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:31 smithi067 ceph-mon[22413]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:31.602 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:31 smithi067 ceph-mon[27137]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:31.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:31 smithi086 ceph-mon[23164]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:33.798 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:33 smithi067 ceph-mon[27137]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:33.799 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:33 smithi067 ceph-mon[22413]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:33.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:33 smithi086 ceph-mon[23164]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:34.810 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:34 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:26:34.810 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:34 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:34.810 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:34 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:26:34.810 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:34 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:34 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:26:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:34 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:35.751 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:35 smithi067 ceph-mon[22413]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:35.751 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:35 smithi067 ceph-mon[22413]: Deploying daemon osd.0 on smithi067 2024-05-06T12:26:35.751 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:35 smithi067 ceph-mon[27137]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:35.751 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:35 smithi067 ceph-mon[27137]: Deploying daemon osd.0 on smithi067 2024-05-06T12:26:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:35 smithi086 ceph-mon[23164]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:35 smithi086 ceph-mon[23164]: Deploying daemon osd.0 on smithi067 2024-05-06T12:26:37.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:37 smithi086 ceph-mon[23164]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:37.974 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:37 smithi067 ceph-mon[22413]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:37.974 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:37 smithi067 ceph-mon[27137]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:39.839 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[27137]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[22413]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:39.840 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:39.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:39 smithi086 ceph-mon[23164]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:39.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:39.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:39.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:39.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:41.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:41 smithi067 ceph-mon[22413]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:41.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:41.723 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:41 smithi067 ceph-mon[27137]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:41.723 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:41.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:41 smithi086 ceph-mon[23164]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:41.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:42.588 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 0 on host 'smithi067' 2024-05-06T12:26:42.886 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:42.886 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:42.886 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: from='osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:26:42.886 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:26:42.886 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:42.886 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:42.886 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: from='osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:42.887 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:43.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:43.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:43.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: from='osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:26:43.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:26:43.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:43.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:26:43.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:43.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:26:43.193 DEBUG:teuthology.orchestra.run.smithi067:osd.0> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.0.service 2024-05-06T12:26:43.195 INFO:tasks.cephadm:Deploying osd.1 on smithi067 with /dev/vg_nvme/lv_3... 2024-05-06T12:26:43.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-06T12:26:44.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[22413]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T12:26:44.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[22413]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T12:26:44.085 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:44.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[22413]: from='osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:26:44.086 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[22413]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:26:44.086 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[27137]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T12:26:44.086 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[27137]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T12:26:44.086 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:44.086 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[27137]: from='osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:26:44.086 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:43 smithi067 ceph-mon[27137]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:26:44.086 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:26:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[29990]: 2024-05-06T12:26:43.738+0000 7f2e296c0700 -1 osd.0 0 waiting for initial osdmap 2024-05-06T12:26:44.087 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:26:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[29990]: 2024-05-06T12:26:43.740+0000 7f2e22053700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:26:44.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:43 smithi086 ceph-mon[23164]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T12:26:44.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:43 smithi086 ceph-mon[23164]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T12:26:44.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:43 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:44.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:43 smithi086 ceph-mon[23164]: from='osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:26:44.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:43 smithi086 ceph-mon[23164]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:26:44.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[22413]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:26:44.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[22413]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T12:26:44.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:44.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:44.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[22413]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:44.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[27137]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:26:44.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[27137]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T12:26:44.857 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:44.857 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:44.857 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:44 smithi067 ceph-mon[27137]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:45.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:44 smithi086 ceph-mon[23164]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:26:45.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:44 smithi086 ceph-mon[23164]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T12:26:45.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:44 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:45.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:44 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:45.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:44 smithi086 ceph-mon[23164]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[27137]: osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353] boot 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[27137]: osdmap e8: 1 total, 1 up, 1 in 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[22413]: osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353] boot 2024-05-06T12:26:46.115 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[22413]: osdmap e8: 1 total, 1 up, 1 in 2024-05-06T12:26:46.116 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:45 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:45 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:26:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:45 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:26:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:45 smithi086 ceph-mon[23164]: osd.0 [v2:172.21.15.67:6802/1842861353,v1:172.21.15.67:6803/1842861353] boot 2024-05-06T12:26:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:45 smithi086 ceph-mon[23164]: osdmap e8: 1 total, 1 up, 1 in 2024-05-06T12:26:46.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:45 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-06T12:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-06T12:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-06T12:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-06T12:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0169299 s, 619 MB/s 2024-05-06T12:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-06T12:26:46.830 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:46 smithi067 ceph-mon[22413]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T12:26:46.831 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:46 smithi067 ceph-mon[22413]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:46.831 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:46 smithi067 ceph-mon[27137]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T12:26:46.831 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:46 smithi067 ceph-mon[27137]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:46.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-05-06T12:26:47.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:46 smithi086 ceph-mon[23164]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T12:26:47.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:46 smithi086 ceph-mon[23164]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:49.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[22413]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:49.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:26:49.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.183 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[27137]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:49.184 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.184 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:26:49.184 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.184 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:49 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:49 smithi086 ceph-mon[23164]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:49.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:49 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:49 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:26:49.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:49 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:49.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:49 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:26:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:50 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi067 2024-05-06T12:26:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:50 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi067 to 16064M 2024-05-06T12:26:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:50 smithi086 ceph-mon[23164]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:50 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:26:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:50 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:26:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:50 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:50.440 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi067 2024-05-06T12:26:50.440 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi067 to 16064M 2024-05-06T12:26:50.440 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[27137]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:50.440 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:26:50.440 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:26:50.440 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:50.440 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi067 2024-05-06T12:26:50.441 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi067 to 16064M 2024-05-06T12:26:50.441 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[22413]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:26:50.441 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:26:50.441 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:26:50.441 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:50 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:51 smithi086 ceph-mon[23164]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:51.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:51 smithi067 ceph-mon[27137]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:51 smithi067 ceph-mon[22413]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:52 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1282700821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88072cfe-96ff-4a30-93db-74402fa65199"}]: dispatch 2024-05-06T12:26:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:52 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1282700821' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88072cfe-96ff-4a30-93db-74402fa65199"}]': finished 2024-05-06T12:26:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:52 smithi086 ceph-mon[23164]: osdmap e10: 2 total, 1 up, 2 in 2024-05-06T12:26:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:52 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:26:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1282700821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88072cfe-96ff-4a30-93db-74402fa65199"}]: dispatch 2024-05-06T12:26:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1282700821' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88072cfe-96ff-4a30-93db-74402fa65199"}]': finished 2024-05-06T12:26:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[22413]: osdmap e10: 2 total, 1 up, 2 in 2024-05-06T12:26:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:26:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1282700821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88072cfe-96ff-4a30-93db-74402fa65199"}]: dispatch 2024-05-06T12:26:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1282700821' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88072cfe-96ff-4a30-93db-74402fa65199"}]': finished 2024-05-06T12:26:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[27137]: osdmap e10: 2 total, 1 up, 2 in 2024-05-06T12:26:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:52 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:26:53.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:53 smithi067 ceph-mon[27137]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:53.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:53 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3599273790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:26:53.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:53 smithi067 ceph-mon[22413]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:53.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:53 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3599273790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:26:53.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:53 smithi086 ceph-mon[23164]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:53.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:53 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3599273790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:26:55.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:55 smithi067 ceph-mon[22413]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:55.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:55 smithi067 ceph-mon[27137]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:55.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:55 smithi086 ceph-mon[23164]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:57.453 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:57 smithi067 ceph-mon[27137]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:57.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:57 smithi067 ceph-mon[22413]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:57.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:57 smithi086 ceph-mon[23164]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:58.391 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:26:58.392 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:58.392 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:26:58.392 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:58.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:26:58.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:26:59.350 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:59 smithi067 ceph-mon[22413]: Deploying daemon osd.1 on smithi067 2024-05-06T12:26:59.350 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:26:59 smithi067 ceph-mon[22413]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:59.350 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:59 smithi067 ceph-mon[27137]: Deploying daemon osd.1 on smithi067 2024-05-06T12:26:59.350 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:26:59 smithi067 ceph-mon[27137]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:26:59.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:59 smithi086 ceph-mon[23164]: Deploying daemon osd.1 on smithi067 2024-05-06T12:26:59.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:26:59 smithi086 ceph-mon[23164]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:01.205 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:01 smithi067 ceph-mon[22413]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:01.205 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:01 smithi067 ceph-mon[27137]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:01.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:01 smithi086 ceph-mon[23164]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:03 smithi086 ceph-mon[23164]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:03 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:03 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:03 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:03 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:03.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[22413]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:03.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:03.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:03.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:03.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:03.424 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[27137]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:03.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:03.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:03.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:03.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:05.420 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:05 smithi067 ceph-mon[22413]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:05.421 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:05 smithi067 ceph-mon[27137]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:05.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:05 smithi086 ceph-mon[23164]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:06.035 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 1 on host 'smithi067' 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:06.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:06.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:06.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:27:06.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:27:06.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:06.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:06.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:06.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:06.675 DEBUG:teuthology.orchestra.run.smithi067:osd.1> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.1.service 2024-05-06T12:27:06.677 INFO:tasks.cephadm:Deploying osd.2 on smithi067 with /dev/vg_nvme/lv_2... 2024-05-06T12:27:06.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-06T12:27:07.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:07 smithi086 ceph-mon[23164]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:07.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:07 smithi086 ceph-mon[23164]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T12:27:07.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:07 smithi086 ceph-mon[23164]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T12:27:07.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:07 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:07.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:07 smithi086 ceph-mon[23164]: from='osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:07.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:07 smithi086 ceph-mon[23164]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:07.556 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[27137]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[27137]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[27137]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[27137]: from='osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[27137]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[22413]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[22413]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[22413]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[22413]: from='osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:07.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:07 smithi067 ceph-mon[22413]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:07.558 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[32953]: 2024-05-06T12:27:07.261+0000 7fbec612a700 -1 osd.1 0 waiting for initial osdmap 2024-05-06T12:27:07.558 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:07 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[32953]: 2024-05-06T12:27:07.264+0000 7fbec1ac3700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:27:08.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[22413]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:08.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[22413]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T12:27:08.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:08.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:08.454 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[27137]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:08.454 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[27137]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T12:27:08.454 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:08.454 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:08 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:08 smithi086 ceph-mon[23164]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:08 smithi086 ceph-mon[23164]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T12:27:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:08 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:08 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[22413]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[22413]: osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866] boot 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[22413]: osdmap e13: 2 total, 2 up, 2 in 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:27:09.560 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[27137]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:09.561 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:09.561 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[27137]: osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866] boot 2024-05-06T12:27:09.561 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[27137]: osdmap e13: 2 total, 2 up, 2 in 2024-05-06T12:27:09.561 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:09 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:27:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:09 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:27:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:09 smithi086 ceph-mon[23164]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T12:27:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:09 smithi086 ceph-mon[23164]: osd.1 [v2:172.21.15.67:6810/4168781866,v1:172.21.15.67:6811/4168781866] boot 2024-05-06T12:27:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:09 smithi086 ceph-mon[23164]: osdmap e13: 2 total, 2 up, 2 in 2024-05-06T12:27:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:27:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T12:27:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi067 2024-05-06T12:27:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:10.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:10.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi067 to 8032M 2024-05-06T12:27:10.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:10 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T12:27:10.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi067 2024-05-06T12:27:10.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi067 to 8032M 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi067 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi067 to 8032M 2024-05-06T12:27:10.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:10 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:10.759 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-06T12:27:10.760 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-06T12:27:10.760 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-06T12:27:10.760 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-06T12:27:10.760 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0133193 s, 787 MB/s 2024-05-06T12:27:10.760 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 2024-05-06T12:27:10.760 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-06T12:27:11.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-05-06T12:27:11.863 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:11 smithi067 ceph-mon[22413]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:11.863 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:11 smithi067 ceph-mon[27137]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:11 smithi086 ceph-mon[23164]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:13 smithi086 ceph-mon[23164]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:13 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:27:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:13 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:27:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:13 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:13.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[22413]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:13.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:27:13.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:27:13.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:13.939 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[27137]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:13.939 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:27:13.940 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:27:13.940 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:14.850 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:14 smithi067 ceph-mon[22413]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:27:14.850 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:14 smithi067 ceph-mon[27137]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:27:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:14 smithi086 ceph-mon[23164]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:27:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:15 smithi086 ceph-mon[23164]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:15 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1743697521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]: dispatch 2024-05-06T12:27:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:15 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]: dispatch 2024-05-06T12:27:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:15 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]': finished 2024-05-06T12:27:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:15 smithi086 ceph-mon[23164]: osdmap e15: 3 total, 2 up, 3 in 2024-05-06T12:27:15.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:15 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:15.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[22413]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1743697521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]: dispatch 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]: dispatch 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]': finished 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[22413]: osdmap e15: 3 total, 2 up, 3 in 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[27137]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1743697521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]: dispatch 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]: dispatch 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "051681d5-efcb-4a36-81b6-266ffb96dfd2"}]': finished 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[27137]: osdmap e15: 3 total, 2 up, 3 in 2024-05-06T12:27:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:15 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:16.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:16 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1881146594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:27:16.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:16 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1881146594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:27:17.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:16 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1881146594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:27:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:17 smithi067 ceph-mon[22413]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:17.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:17 smithi067 ceph-mon[27137]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:18.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:17 smithi086 ceph-mon[23164]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:19.794 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:19 smithi067 ceph-mon[22413]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:19.794 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:19 smithi067 ceph-mon[27137]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:20.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:19 smithi086 ceph-mon[23164]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:21.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:21 smithi067 ceph-mon[22413]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:21.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:21 smithi067 ceph-mon[27137]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:22.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:21 smithi086 ceph-mon[23164]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:22 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:27:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:22 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:22.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:22 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:27:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:22 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:23.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:22 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:27:23.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:22 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:23.963 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:23 smithi067 ceph-mon[22413]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:23.963 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:23 smithi067 ceph-mon[22413]: Deploying daemon osd.2 on smithi067 2024-05-06T12:27:23.963 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:23 smithi067 ceph-mon[27137]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:23.963 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:23 smithi067 ceph-mon[27137]: Deploying daemon osd.2 on smithi067 2024-05-06T12:27:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:23 smithi086 ceph-mon[23164]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:23 smithi086 ceph-mon[23164]: Deploying daemon osd.2 on smithi067 2024-05-06T12:27:25.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:25 smithi067 ceph-mon[27137]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:25.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:25 smithi067 ceph-mon[22413]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:26.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:25 smithi086 ceph-mon[23164]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:27:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:27:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:27:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:27:27.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:27:27.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:27:27.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[27137]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:27.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[22413]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:27.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:27 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:27 smithi086 ceph-mon[23164]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:27 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:28.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:27 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:28.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:27 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:28.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:27 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:29.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:29 smithi067 ceph-mon[22413]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:29.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:29 smithi067 ceph-mon[27137]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:30.055 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 2 on host 'smithi067' 2024-05-06T12:27:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:29 smithi086 ceph-mon[23164]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:30.658 DEBUG:teuthology.orchestra.run.smithi067:osd.2> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.2.service 2024-05-06T12:27:30.661 INFO:tasks.cephadm:Deploying osd.3 on smithi067 with /dev/vg_nvme/lv_1... 2024-05-06T12:27:30.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[22413]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[22413]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[27137]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[27137]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:27:31.167 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:31 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:31 smithi086 ceph-mon[23164]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:31 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:31 smithi086 ceph-mon[23164]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:27:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:31 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:32.359 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[22413]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T12:27:32.359 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[22413]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T12:27:32.359 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:32.359 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[22413]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:32.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[27137]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T12:27:32.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[27137]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T12:27:32.359 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:32.360 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:32 smithi067 ceph-mon[27137]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:32.360 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[35890]: 2024-05-06T12:27:32.065+0000 7f0facb6e700 -1 osd.2 0 waiting for initial osdmap 2024-05-06T12:27:32.360 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[35890]: 2024-05-06T12:27:32.068+0000 7f0fa7d06700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:27:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:32 smithi086 ceph-mon[23164]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T12:27:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:32 smithi086 ceph-mon[23164]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T12:27:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:32 smithi086 ceph-mon[23164]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[22413]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[22413]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[22413]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[27137]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[27137]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[27137]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:33.179 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:33 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:33 smithi086 ceph-mon[23164]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T12:27:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:33 smithi086 ceph-mon[23164]: from='osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:33 smithi086 ceph-mon[23164]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T12:27:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:33 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:33 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-06T12:27:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-06T12:27:33.493 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-06T12:27:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-06T12:27:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127956 s, 819 MB/s 2024-05-06T12:27:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-06T12:27:34.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_1 2024-05-06T12:27:34.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:27:34.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:27:34.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[22413]: osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293] boot 2024-05-06T12:27:34.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[22413]: osdmap e18: 3 total, 3 up, 3 in 2024-05-06T12:27:34.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:27:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:27:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[27137]: osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293] boot 2024-05-06T12:27:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[27137]: osdmap e18: 3 total, 3 up, 3 in 2024-05-06T12:27:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:34 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:34 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:27:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:34 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:27:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:34 smithi086 ceph-mon[23164]: osd.2 [v2:172.21.15.67:6818/4011218293,v1:172.21.15.67:6819/4011218293] boot 2024-05-06T12:27:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:34 smithi086 ceph-mon[23164]: osdmap e18: 3 total, 3 up, 3 in 2024-05-06T12:27:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:34 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:27:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:35 smithi086 ceph-mon[23164]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:35 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-06T12:27:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:35 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-06T12:27:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:35 smithi086 ceph-mon[23164]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T12:27:35.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:35 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T12:27:35.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[22413]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[22413]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[27137]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[27137]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T12:27:35.973 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:35 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T12:27:36.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:36 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T12:27:36.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:36 smithi067 ceph-mon[22413]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T12:27:36.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:36 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T12:27:36.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:36 smithi067 ceph-mon[27137]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T12:27:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:36 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T12:27:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:36 smithi086 ceph-mon[23164]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T12:27:37.100 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:27:36 smithi067 sudo[37238]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T12:27:37.100 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:27:36 smithi067 sudo[37238]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:37.100 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:27:36 smithi067 sudo[37238]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:37.379 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:27:37 smithi067 sudo[37300]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T12:27:37.379 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:27:37 smithi067 sudo[37300]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:37.379 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:27:37 smithi067 sudo[37300]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:37.664 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:37.666 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:37.667 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.667 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:37 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.770 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:37 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:37.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:37 smithi067 sudo[37321]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T12:27:37.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:37 smithi067 sudo[37321]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:37.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:37 smithi067 sudo[37321]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:38.451 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:38 smithi067 sudo[37339]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T12:27:38.451 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:38 smithi067 sudo[37339]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:38.452 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:27:38 smithi067 sudo[37339]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:38 smithi067 ceph-mon[22413]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:27:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:38 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi067 2024-05-06T12:27:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:38 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi067 to 5354M 2024-05-06T12:27:38.725 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:38 smithi067 sudo[37355]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T12:27:38.726 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:38 smithi067 sudo[37355]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:38.726 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:38 smithi067 sudo[37355]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:38.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:38 smithi067 ceph-mon[27137]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:27:38.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:38 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi067 2024-05-06T12:27:38.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:38 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi067 to 5354M 2024-05-06T12:27:38.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:38 smithi086 ceph-mon[23164]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:27:38.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:38 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi067 2024-05-06T12:27:38.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:38 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi067 to 5354M 2024-05-06T12:27:39.465 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37391]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T12:27:39.465 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37391]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:39.465 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37391]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:39.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37395]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T12:27:39.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37395]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:39.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37395]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37398]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T12:27:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37398]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 sudo[37398]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 ceph-mon[22413]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:39 smithi067 ceph-mon[22413]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-06T12:27:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:39 smithi086 ceph-mon[23164]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:39 smithi086 ceph-mon[23164]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-06T12:27:40.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 sudo[37435]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T12:27:40.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 sudo[37435]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:40.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 sudo[37435]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 sudo[37486]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T12:27:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 sudo[37486]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 sudo[37486]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 ceph-mon[27137]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:40 smithi067 ceph-mon[27137]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-06T12:27:41.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 sudo[24691]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T12:27:41.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 sudo[24691]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:41.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 sudo[24691]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:41.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 sudo[24694]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T12:27:41.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 sudo[24694]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T12:27:41.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 sudo[24694]: pam_unix(sudo:session): session closed for user root 2024-05-06T12:27:41.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2591053587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315d956c-b263-46c8-97b3-5346b0d05d11"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2591053587' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315d956c-b263-46c8-97b3-5346b0d05d11"}]': finished 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: osdmap e22: 4 total, 3 up, 4 in 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2930461911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:27:41.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:41.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2591053587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315d956c-b263-46c8-97b3-5346b0d05d11"}]: dispatch 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2591053587' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315d956c-b263-46c8-97b3-5346b0d05d11"}]': finished 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: osdmap e22: 4 total, 3 up, 4 in 2024-05-06T12:27:41.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2930461911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2591053587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315d956c-b263-46c8-97b3-5346b0d05d11"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2591053587' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315d956c-b263-46c8-97b3-5346b0d05d11"}]': finished 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: osdmap e22: 4 total, 3 up, 4 in 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2930461911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:27:41.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:27:42.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:42 smithi086 ceph-mon[23164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:42.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:42 smithi086 ceph-mon[23164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:42.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:42.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:42 smithi067 ceph-mon[22413]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:42.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:42 smithi067 ceph-mon[22413]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:42.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:42.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:42 smithi067 ceph-mon[27137]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T12:27:42.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:42 smithi067 ceph-mon[27137]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T12:27:42.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:43.459 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:43 smithi067 ceph-mon[27137]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:43.459 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:43 smithi067 ceph-mon[22413]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:43 smithi086 ceph-mon[23164]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:45.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:45 smithi067 ceph-mon[22413]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:45.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:45 smithi067 ceph-mon[27137]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:45.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:45 smithi086 ceph-mon[23164]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:47 smithi086 ceph-mon[23164]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 129 KiB/s wr, 9 op/s 2024-05-06T12:27:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:27:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:47 smithi086 ceph-mon[23164]: Deploying daemon osd.3 on smithi067 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[22413]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 129 KiB/s wr, 9 op/s 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[22413]: Deploying daemon osd.3 on smithi067 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[27137]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 129 KiB/s wr, 9 op/s 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:47.667 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:47 smithi067 ceph-mon[27137]: Deploying daemon osd.3 on smithi067 2024-05-06T12:27:49.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:49 smithi067 ceph-mon[22413]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 97 KiB/s wr, 6 op/s 2024-05-06T12:27:49.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:49 smithi067 ceph-mon[27137]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 97 KiB/s wr, 6 op/s 2024-05-06T12:27:49.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:49 smithi086 ceph-mon[23164]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 97 KiB/s wr, 6 op/s 2024-05-06T12:27:51.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:50 smithi067 ceph-mon[22413]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-06T12:27:51.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:50 smithi067 ceph-mon[27137]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-06T12:27:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:50 smithi086 ceph-mon[23164]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-06T12:27:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:52 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:52 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:52 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:52 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:52 smithi086 ceph-mon[23164]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-06T12:27:53.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:53.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:53.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:53.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:53.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[22413]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-06T12:27:53.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:53.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:53.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:53.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:53.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:52 smithi067 ceph-mon[27137]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-06T12:27:55.358 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 3 on host 'smithi067' 2024-05-06T12:27:55.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-06T12:27:55.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:27:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:27:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:55 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:55.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:27:55.938 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:55 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:27:55.960 DEBUG:teuthology.orchestra.run.smithi067:osd.3> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.3.service 2024-05-06T12:27:55.963 INFO:tasks.cephadm:Deploying osd.4 on smithi086 with /dev/vg_nvme/lv_4... 2024-05-06T12:27:55.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-06T12:27:57.126 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[22413]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T12:27:57.126 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[22413]: osdmap e23: 4 total, 3 up, 4 in 2024-05-06T12:27:57.126 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:57.126 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[22413]: from='osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:57.126 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[22413]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:57.126 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[22413]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:57.127 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[27137]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T12:27:57.127 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[27137]: osdmap e23: 4 total, 3 up, 4 in 2024-05-06T12:27:57.127 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:57.127 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[27137]: from='osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:57.127 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[27137]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:57.127 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:56 smithi067 ceph-mon[27137]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:57.127 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:27:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[38867]: 2024-05-06T12:27:56.937+0000 7fc397029700 -1 osd.3 0 waiting for initial osdmap 2024-05-06T12:27:57.127 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:27:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[38867]: 2024-05-06T12:27:56.942+0000 7fc3921c1700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:27:57.207 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:56 smithi086 ceph-mon[23164]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T12:27:57.208 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:56 smithi086 ceph-mon[23164]: osdmap e23: 4 total, 3 up, 4 in 2024-05-06T12:27:57.208 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:56 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:57.208 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:56 smithi086 ceph-mon[23164]: from='osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:57.208 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:56 smithi086 ceph-mon[23164]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:27:57.208 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:56 smithi086 ceph-mon[23164]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:57.938 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-06T12:27:57.938 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-06T12:27:57.938 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-06T12:27:57.938 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-06T12:27:57.938 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114285 s, 918 MB/s 2024-05-06T12:27:57.938 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 2024-05-06T12:27:57.939 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-06T12:27:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[22413]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[22413]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T12:27:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[27137]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[27137]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T12:27:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:57 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:58.286 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:57 smithi086 ceph-mon[23164]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-06T12:27:58.286 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:57 smithi086 ceph-mon[23164]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T12:27:58.286 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:57 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:58.286 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:57 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:58.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118] boot 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: osdmap e25: 4 total, 4 up, 4 in 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:58 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:27:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118] boot 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: osdmap e25: 4 total, 4 up, 4 in 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: osd.3 [v2:172.21.15.67:6826/3041731118,v1:172.21.15.67:6827/3041731118] boot 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: osdmap e25: 4 total, 4 up, 4 in 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:27:59.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:58 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:00.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:59 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi067 2024-05-06T12:28:00.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:59 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi067 to 4016M 2024-05-06T12:28:00.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:27:59 smithi086 ceph-mon[23164]: osdmap e26: 4 total, 4 up, 4 in 2024-05-06T12:28:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:59 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi067 2024-05-06T12:28:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:59 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi067 to 4016M 2024-05-06T12:28:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:27:59 smithi067 ceph-mon[22413]: osdmap e26: 4 total, 4 up, 4 in 2024-05-06T12:28:00.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:59 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi067 2024-05-06T12:28:00.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:59 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi067 to 4016M 2024-05-06T12:28:00.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:27:59 smithi067 ceph-mon[27137]: osdmap e26: 4 total, 4 up, 4 in 2024-05-06T12:28:01.100 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:00 smithi086 ceph-mon[23164]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:01.100 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:00 smithi086 ceph-mon[23164]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:01.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:00 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:01.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:00 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:01.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:00 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[22413]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[22413]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:01.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[27137]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[27137]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:00 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:03 smithi086 ceph-mon[23164]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:03 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/2072853656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]: dispatch 2024-05-06T12:28:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:03 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]: dispatch 2024-05-06T12:28:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:03 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]': finished 2024-05-06T12:28:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:03 smithi086 ceph-mon[23164]: osdmap e27: 5 total, 4 up, 5 in 2024-05-06T12:28:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:03 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[22413]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/2072853656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]: dispatch 2024-05-06T12:28:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]: dispatch 2024-05-06T12:28:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]': finished 2024-05-06T12:28:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[22413]: osdmap e27: 5 total, 4 up, 5 in 2024-05-06T12:28:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:03.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[27137]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:03.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/2072853656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]: dispatch 2024-05-06T12:28:03.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]: dispatch 2024-05-06T12:28:03.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52e8b1fc-7a3c-4c99-81cd-c690bc506e4a"}]': finished 2024-05-06T12:28:03.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[27137]: osdmap e27: 5 total, 4 up, 5 in 2024-05-06T12:28:03.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:04 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/3238817320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:04 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/3238817320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:04 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/3238817320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:05 smithi086 ceph-mon[23164]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:05 smithi067 ceph-mon[22413]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:05.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:05 smithi067 ceph-mon[27137]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:07.292 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:07 smithi086 ceph-mon[23164]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:07.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:07 smithi067 ceph-mon[22413]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:07.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:07 smithi067 ceph-mon[27137]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:09.310 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:09 smithi086 ceph-mon[23164]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:09.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:28:09.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:09.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:09 smithi086 ceph-mon[23164]: Deploying daemon osd.4 on smithi086 2024-05-06T12:28:09.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[27137]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:09.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:28:09.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:09.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[27137]: Deploying daemon osd.4 on smithi086 2024-05-06T12:28:09.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[22413]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:09.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:28:09.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:09.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:09 smithi067 ceph-mon[22413]: Deploying daemon osd.4 on smithi086 2024-05-06T12:28:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:10 smithi067 ceph-mon[22413]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:11.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:10 smithi067 ceph-mon[27137]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:11.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:10 smithi086 ceph-mon[23164]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:13.231 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:13 smithi086 ceph-mon[23164]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:13.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:13 smithi067 ceph-mon[22413]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:13.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:13 smithi067 ceph-mon[27137]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:14 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:14 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:14 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:14.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:14 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:14.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:14.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:14.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:14.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:14 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:15.684 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:15 smithi086 ceph-mon[23164]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:15.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:15 smithi067 ceph-mon[22413]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:15.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:15 smithi067 ceph-mon[27137]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:16.887 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 4 on host 'smithi086' 2024-05-06T12:28:17.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:17.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:28:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:28:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:16 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:17.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:17.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:28:17.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:28:17.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:17.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:17.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:16 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:17.529 DEBUG:teuthology.orchestra.run.smithi086:osd.4> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.4.service 2024-05-06T12:28:17.531 INFO:tasks.cephadm:Deploying osd.5 on smithi086 with /dev/vg_nvme/lv_3... 2024-05-06T12:28:17.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-06T12:28:18.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:18 smithi086 ceph-mon[23164]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T12:28:18.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:18 smithi086 ceph-mon[23164]: osdmap e28: 5 total, 4 up, 5 in 2024-05-06T12:28:18.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:18 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:18.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:18 smithi086 ceph-mon[23164]: from='osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:18.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:18 smithi086 ceph-mon[23164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:18.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:28:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[26532]: 2024-05-06T12:28:18.286+0000 7f8e53706700 -1 osd.4 0 waiting for initial osdmap 2024-05-06T12:28:18.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:28:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[26532]: 2024-05-06T12:28:18.290+0000 7f8e4f09f700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:28:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[22413]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T12:28:18.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[22413]: osdmap e28: 5 total, 4 up, 5 in 2024-05-06T12:28:18.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:18.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[22413]: from='osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:18.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[22413]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:18.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[27137]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T12:28:18.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[27137]: osdmap e28: 5 total, 4 up, 5 in 2024-05-06T12:28:18.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:18.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[27137]: from='osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:18.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:18 smithi067 ceph-mon[27137]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:19.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:19 smithi086 ceph-mon[23164]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:19.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:19 smithi086 ceph-mon[23164]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:28:19.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:19 smithi086 ceph-mon[23164]: osdmap e29: 5 total, 4 up, 5 in 2024-05-06T12:28:19.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:19 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:19.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:19 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:19.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[22413]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:19.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[22413]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:28:19.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[22413]: osdmap e29: 5 total, 4 up, 5 in 2024-05-06T12:28:19.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:19.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:19.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[27137]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T12:28:19.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[27137]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:28:19.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[27137]: osdmap e29: 5 total, 4 up, 5 in 2024-05-06T12:28:19.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:19.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:19 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:20.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:20 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:28:20.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:20 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:28:20.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:20 smithi086 ceph-mon[23164]: osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317] boot 2024-05-06T12:28:20.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:20 smithi086 ceph-mon[23164]: osdmap e30: 5 total, 5 up, 5 in 2024-05-06T12:28:20.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:20 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:20.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:20 smithi086 ceph-mon[23164]: osdmap e31: 5 total, 5 up, 5 in 2024-05-06T12:28:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:28:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:28:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[22413]: osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317] boot 2024-05-06T12:28:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[22413]: osdmap e30: 5 total, 5 up, 5 in 2024-05-06T12:28:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[22413]: osdmap e31: 5 total, 5 up, 5 in 2024-05-06T12:28:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:28:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:28:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[27137]: osd.4 [v2:172.21.15.86:6800/1658830317,v1:172.21.15.86:6801/1658830317] boot 2024-05-06T12:28:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[27137]: osdmap e30: 5 total, 5 up, 5 in 2024-05-06T12:28:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:28:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:20 smithi067 ceph-mon[27137]: osdmap e31: 5 total, 5 up, 5 in 2024-05-06T12:28:20.779 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-06T12:28:20.780 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-06T12:28:20.780 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-06T12:28:20.780 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-06T12:28:20.780 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124113 s, 845 MB/s 2024-05-06T12:28:20.780 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-06T12:28:21.458 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:21 smithi086 ceph-mon[23164]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:21.458 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:21 smithi086 ceph-mon[23164]: osdmap e32: 5 total, 5 up, 5 in 2024-05-06T12:28:21.626 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:21 smithi067 ceph-mon[22413]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:21.627 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:21 smithi067 ceph-mon[22413]: osdmap e32: 5 total, 5 up, 5 in 2024-05-06T12:28:21.627 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:21 smithi067 ceph-mon[27137]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:21.627 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:21 smithi067 ceph-mon[27137]: osdmap e32: 5 total, 5 up, 5 in 2024-05-06T12:28:21.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-05-06T12:28:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:22 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi086 2024-05-06T12:28:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:22 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:22 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:22 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi086 to 17088M 2024-05-06T12:28:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:22 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:22 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:22 smithi086 ceph-mon[23164]: osdmap e33: 5 total, 5 up, 5 in 2024-05-06T12:28:22.837 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi086 2024-05-06T12:28:22.837 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.837 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:22.837 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi086 to 17088M 2024-05-06T12:28:22.837 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.837 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[27137]: osdmap e33: 5 total, 5 up, 5 in 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi086 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi086 to 17088M 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:22.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:22 smithi067 ceph-mon[22413]: osdmap e33: 5 total, 5 up, 5 in 2024-05-06T12:28:23.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:23 smithi086 ceph-mon[23164]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:23.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:23 smithi086 ceph-mon[23164]: osdmap e34: 5 total, 5 up, 5 in 2024-05-06T12:28:23.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:23 smithi067 ceph-mon[22413]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:23.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:23 smithi067 ceph-mon[22413]: osdmap e34: 5 total, 5 up, 5 in 2024-05-06T12:28:23.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:23 smithi067 ceph-mon[27137]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:23.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:23 smithi067 ceph-mon[27137]: osdmap e34: 5 total, 5 up, 5 in 2024-05-06T12:28:24.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:24.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:24.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:24 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:24.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:24.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:24.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:24 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:24.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:24.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:24.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:24 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:25 smithi086 ceph-mon[23164]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:25 smithi086 ceph-mon[23164]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:25 smithi067 ceph-mon[22413]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:25 smithi067 ceph-mon[22413]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:25.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:25 smithi067 ceph-mon[27137]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:25.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:25 smithi067 ceph-mon[27137]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:26 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/3189928890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]: dispatch 2024-05-06T12:28:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:26 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]: dispatch 2024-05-06T12:28:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:26 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]': finished 2024-05-06T12:28:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:26 smithi086 ceph-mon[23164]: osdmap e35: 6 total, 5 up, 6 in 2024-05-06T12:28:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:28:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:28:26.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/3189928890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]: dispatch 2024-05-06T12:28:26.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]: dispatch 2024-05-06T12:28:26.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]': finished 2024-05-06T12:28:26.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[22413]: osdmap e35: 6 total, 5 up, 6 in 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/3189928890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]: dispatch 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]: dispatch 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5df6cce9-b99e-407c-8d88-ed2cd6f136be"}]': finished 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[27137]: osdmap e35: 6 total, 5 up, 6 in 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:28:26.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:28:27.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:27 smithi086 ceph-mon[23164]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-06T12:28:27.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:27 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/1290266265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:27.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:27 smithi067 ceph-mon[22413]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-06T12:28:27.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:27 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/1290266265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:27.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:27 smithi067 ceph-mon[27137]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-06T12:28:27.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:27 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/1290266265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:29.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:29 smithi086 ceph-mon[23164]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-06T12:28:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:29 smithi067 ceph-mon[22413]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-06T12:28:29.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:29 smithi067 ceph-mon[27137]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-06T12:28:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:30 smithi067 ceph-mon[22413]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T12:28:31.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:30 smithi067 ceph-mon[27137]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T12:28:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:30 smithi086 ceph-mon[23164]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T12:28:32.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:28:32.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:32.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:28:32.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:28:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:33.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:33 smithi067 ceph-mon[22413]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-06T12:28:33.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:33 smithi067 ceph-mon[22413]: Deploying daemon osd.5 on smithi086 2024-05-06T12:28:33.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:33 smithi067 ceph-mon[27137]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-06T12:28:33.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:33 smithi067 ceph-mon[27137]: Deploying daemon osd.5 on smithi086 2024-05-06T12:28:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:33 smithi086 ceph-mon[23164]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-06T12:28:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:33 smithi086 ceph-mon[23164]: Deploying daemon osd.5 on smithi086 2024-05-06T12:28:35.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:35 smithi086 ceph-mon[23164]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T12:28:35.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:35 smithi067 ceph-mon[22413]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T12:28:35.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:35 smithi067 ceph-mon[27137]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T12:28:37.203 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:37 smithi086 ceph-mon[23164]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:37.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:37 smithi067 ceph-mon[22413]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:37.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:37 smithi067 ceph-mon[27137]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:38.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:38.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:38.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:38.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:38 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:38.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:38 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:38.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:38 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:38.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:38 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:38.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:38 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:39 smithi067 ceph-mon[22413]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:39.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:39 smithi067 ceph-mon[27137]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:39.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:39 smithi086 ceph-mon[23164]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:39.914 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 5 on host 'smithi086' 2024-05-06T12:28:40.524 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.5.service 2024-05-06T12:28:40.527 INFO:tasks.cephadm:Deploying osd.6 on smithi086 with /dev/vg_nvme/lv_2... 2024-05-06T12:28:40.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-06T12:28:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[22413]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[22413]: from='osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:28:41.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[22413]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:28:41.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:41.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[27137]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:41.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[27137]: from='osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:28:41.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:40 smithi067 ceph-mon[27137]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:28:41.232 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:40 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:41.233 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:40 smithi086 ceph-mon[23164]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:41.233 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:40 smithi086 ceph-mon[23164]: from='osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:28:41.233 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:40 smithi086 ceph-mon[23164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:28:42.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:41 smithi086 ceph-mon[23164]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T12:28:42.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:41 smithi086 ceph-mon[23164]: osdmap e36: 6 total, 5 up, 6 in 2024-05-06T12:28:42.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:41 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:42.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:41 smithi086 ceph-mon[23164]: from='osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:42.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:41 smithi086 ceph-mon[23164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:42.216 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:28:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[29475]: 2024-05-06T12:28:41.951+0000 7fb535d4f700 -1 osd.5 0 waiting for initial osdmap 2024-05-06T12:28:42.216 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:28:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[29475]: 2024-05-06T12:28:41.956+0000 7fb5306e6700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:28:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[22413]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T12:28:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[22413]: osdmap e36: 6 total, 5 up, 6 in 2024-05-06T12:28:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[22413]: from='osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[22413]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:42.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[27137]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T12:28:42.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[27137]: osdmap e36: 6 total, 5 up, 6 in 2024-05-06T12:28:42.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:42.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[27137]: from='osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:42.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:41 smithi067 ceph-mon[27137]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:28:42.696 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-06T12:28:42.696 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-06T12:28:42.696 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-06T12:28:42.696 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-06T12:28:42.697 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128625 s, 815 MB/s 2024-05-06T12:28:42.697 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-06T12:28:43.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[22413]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:28:43.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[22413]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T12:28:43.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:43.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:43.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[22413]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:43.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:43.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[27137]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:28:43.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[27137]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T12:28:43.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:43.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:43.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[27137]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:43.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:42 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:43.327 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:42 smithi086 ceph-mon[23164]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:28:43.327 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:42 smithi086 ceph-mon[23164]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T12:28:43.328 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:43.328 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:43.328 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:42 smithi086 ceph-mon[23164]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T12:28:43.328 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:42 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:43.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-05-06T12:28:44.224 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:43 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:28:44.224 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:43 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:28:44.224 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:43 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:43 smithi086 ceph-mon[23164]: osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500] boot 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:43 smithi086 ceph-mon[23164]: osdmap e38: 6 total, 6 up, 6 in 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:43 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[27137]: osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500] boot 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[27137]: osdmap e38: 6 total, 6 up, 6 in 2024-05-06T12:28:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:28:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:28:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[22413]: osd.5 [v2:172.21.15.86:6808/2497673500,v1:172.21.15.86:6809/2497673500] boot 2024-05-06T12:28:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[22413]: osdmap e38: 6 total, 6 up, 6 in 2024-05-06T12:28:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:43 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:28:45.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:44 smithi067 ceph-mon[22413]: osdmap e39: 6 total, 6 up, 6 in 2024-05-06T12:28:45.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:44 smithi067 ceph-mon[22413]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:45.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:44 smithi067 ceph-mon[27137]: osdmap e39: 6 total, 6 up, 6 in 2024-05-06T12:28:45.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:44 smithi067 ceph-mon[27137]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:45.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:44 smithi086 ceph-mon[23164]: osdmap e39: 6 total, 6 up, 6 in 2024-05-06T12:28:45.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:44 smithi086 ceph-mon[23164]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:46.246 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:45 smithi086 ceph-mon[23164]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T12:28:46.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:45 smithi067 ceph-mon[22413]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T12:28:46.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:45 smithi067 ceph-mon[27137]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T12:28:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:46 smithi086 ceph-mon[23164]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T12:28:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:46 smithi086 ceph-mon[23164]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:46 smithi086 ceph-mon[23164]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:46 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:46 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:46 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[27137]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T12:28:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[27137]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[27137]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[22413]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T12:28:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[22413]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[22413]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:28:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:28:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:28:47.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:46 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:48.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T12:28:48.318 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:48.319 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:47 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T12:28:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:48.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:28:48.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:48.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:28:48.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:47 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:28:49.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:48 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi086 2024-05-06T12:28:49.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:48 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi086 to 8544M 2024-05-06T12:28:49.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:48 smithi086 ceph-mon[23164]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:49.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:48 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi086 2024-05-06T12:28:49.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:48 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi086 to 8544M 2024-05-06T12:28:49.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:48 smithi067 ceph-mon[22413]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:49.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:48 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi086 2024-05-06T12:28:49.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:48 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi086 to 8544M 2024-05-06T12:28:49.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:48 smithi067 ceph-mon[27137]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:49 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/2079121084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]: dispatch 2024-05-06T12:28:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:49 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]: dispatch 2024-05-06T12:28:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:49 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]': finished 2024-05-06T12:28:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:49 smithi086 ceph-mon[23164]: osdmap e43: 7 total, 6 up, 7 in 2024-05-06T12:28:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:49 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:28:50.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:49 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/3461386937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/2079121084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]: dispatch 2024-05-06T12:28:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]: dispatch 2024-05-06T12:28:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]': finished 2024-05-06T12:28:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[22413]: osdmap e43: 7 total, 6 up, 7 in 2024-05-06T12:28:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:28:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/3461386937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/2079121084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]: dispatch 2024-05-06T12:28:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]: dispatch 2024-05-06T12:28:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b83b846-b7f8-45e9-aecd-1be2a6157b54"}]': finished 2024-05-06T12:28:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[27137]: osdmap e43: 7 total, 6 up, 7 in 2024-05-06T12:28:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:28:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:49 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/3461386937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:28:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:50 smithi086 ceph-mon[23164]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:51.376 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:50 smithi067 ceph-mon[27137]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:51.376 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:50 smithi067 ceph-mon[22413]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:53.282 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:53 smithi086 ceph-mon[23164]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:53.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:53 smithi067 ceph-mon[22413]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:53.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:53 smithi067 ceph-mon[27137]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:55.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:55 smithi086 ceph-mon[23164]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:55.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:55 smithi067 ceph-mon[22413]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:55.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:55 smithi067 ceph-mon[27137]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:56 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:28:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:56 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:56 smithi086 ceph-mon[23164]: Deploying daemon osd.6 on smithi086 2024-05-06T12:28:56.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:56 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:28:56.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:56 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:56.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:56 smithi067 ceph-mon[22413]: Deploying daemon osd.6 on smithi086 2024-05-06T12:28:56.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:56 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:28:56.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:56 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:28:56.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:56 smithi067 ceph-mon[27137]: Deploying daemon osd.6 on smithi086 2024-05-06T12:28:57.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:57 smithi086 ceph-mon[23164]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:57.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:57 smithi067 ceph-mon[22413]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:57.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:57 smithi067 ceph-mon[27137]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:59.328 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:28:59 smithi086 ceph-mon[23164]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:59.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:28:59 smithi067 ceph-mon[22413]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:28:59.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:28:59 smithi067 ceph-mon[27137]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:01.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:01 smithi086 ceph-mon[23164]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:01.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:01 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:01.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:01 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:01.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:01 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:01.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:01 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:01.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[27137]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:01.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:01.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:01.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:01.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:01.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[22413]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:01.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:01.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:01.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:01.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:01 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:03.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:03 smithi086 ceph-mon[23164]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:03.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:03 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:03.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:03 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 6 on host 'smithi086' 2024-05-06T12:29:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:03 smithi067 ceph-mon[22413]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:03 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:03.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:03 smithi067 ceph-mon[27137]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:03.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:03.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:03 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:04.573 DEBUG:teuthology.orchestra.run.smithi086:osd.6> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.6.service 2024-05-06T12:29:04.576 INFO:tasks.cephadm:Deploying osd.7 on smithi086 with /dev/vg_nvme/lv_1... 2024-05-06T12:29:04.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-06T12:29:04.861 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:04 smithi086 ceph-mon[23164]: from='osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:29:04.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:04 smithi086 ceph-mon[23164]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:29:04.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:04 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:04.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:04 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:04.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:04 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:04.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:04 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:04.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[27137]: from='osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:29:04.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[27137]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:29:04.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:04.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:04.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:04.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:04.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[22413]: from='osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:29:04.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[22413]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:29:04.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:04.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:04.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:04.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:04 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:05.724 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:05 smithi086 ceph-mon[23164]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:05.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:05 smithi086 ceph-mon[23164]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T12:29:05.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:05 smithi086 ceph-mon[23164]: osdmap e44: 7 total, 6 up, 7 in 2024-05-06T12:29:05.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:05 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:05.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:05 smithi086 ceph-mon[23164]: from='osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:05.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:05 smithi086 ceph-mon[23164]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:05.725 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:29:05 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[32364]: 2024-05-06T12:29:05.556+0000 7f9b48f2f700 -1 osd.6 0 waiting for initial osdmap 2024-05-06T12:29:05.725 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:29:05 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[32364]: 2024-05-06T12:29:05.564+0000 7f9b438c6700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:29:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[22413]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[22413]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T12:29:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[22413]: osdmap e44: 7 total, 6 up, 7 in 2024-05-06T12:29:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[22413]: from='osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[22413]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[27137]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[27137]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[27137]: osdmap e44: 7 total, 6 up, 7 in 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[27137]: from='osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:05 smithi067 ceph-mon[27137]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:06.828 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:06 smithi086 ceph-mon[23164]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:29:06.828 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:06 smithi086 ceph-mon[23164]: osdmap e45: 7 total, 6 up, 7 in 2024-05-06T12:29:06.828 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:06.828 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:06 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:06.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[22413]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:29:06.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[22413]: osdmap e45: 7 total, 6 up, 7 in 2024-05-06T12:29:06.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:06.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:06.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[27137]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:29:06.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[27137]: osdmap e45: 7 total, 6 up, 7 in 2024-05-06T12:29:06.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:06.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:06 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-06T12:29:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-06T12:29:07.610 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-06T12:29:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-06T12:29:07.610 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125501 s, 836 MB/s 2024-05-06T12:29:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-06T12:29:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:07 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:29:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:07 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:29:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:07 smithi086 ceph-mon[23164]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:07 smithi086 ceph-mon[23164]: osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570] boot 2024-05-06T12:29:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:07 smithi086 ceph-mon[23164]: osdmap e46: 7 total, 7 up, 7 in 2024-05-06T12:29:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:07 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:07.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:29:07.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:29:07.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[22413]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[22413]: osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570] boot 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[22413]: osdmap e46: 7 total, 7 up, 7 in 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[27137]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[27137]: osd.6 [v2:172.21.15.86:6816/3070209570,v1:172.21.15.86:6817/3070209570] boot 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[27137]: osdmap e46: 7 total, 7 up, 7 in 2024-05-06T12:29:07.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:07 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:08.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2024-05-06T12:29:08.834 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:08 smithi086 ceph-mon[23164]: osdmap e47: 7 total, 7 up, 7 in 2024-05-06T12:29:08.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:08 smithi067 ceph-mon[22413]: osdmap e47: 7 total, 7 up, 7 in 2024-05-06T12:29:08.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:08 smithi067 ceph-mon[27137]: osdmap e47: 7 total, 7 up, 7 in 2024-05-06T12:29:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: osdmap e48: 7 total, 7 up, 7 in 2024-05-06T12:29:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:09 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: osdmap e48: 7 total, 7 up, 7 in 2024-05-06T12:29:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: osdmap e48: 7 total, 7 up, 7 in 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:09.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:09 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:10.835 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:10 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi086 2024-05-06T12:29:10.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:10 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi086 to 5696M 2024-05-06T12:29:10.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:10 smithi086 ceph-mon[23164]: osdmap e49: 7 total, 7 up, 7 in 2024-05-06T12:29:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:10 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi086 2024-05-06T12:29:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:10 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi086 to 5696M 2024-05-06T12:29:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:10 smithi067 ceph-mon[22413]: osdmap e49: 7 total, 7 up, 7 in 2024-05-06T12:29:10.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:10 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi086 2024-05-06T12:29:10.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:10 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi086 to 5696M 2024-05-06T12:29:10.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:10 smithi067 ceph-mon[27137]: osdmap e49: 7 total, 7 up, 7 in 2024-05-06T12:29:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:11 smithi086 ceph-mon[23164]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:11 smithi086 ceph-mon[23164]: osdmap e50: 7 total, 7 up, 7 in 2024-05-06T12:29:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:11 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:29:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:11 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:29:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:11 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[22413]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[22413]: osdmap e50: 7 total, 7 up, 7 in 2024-05-06T12:29:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:29:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:29:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:11.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[27137]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:11.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[27137]: osdmap e50: 7 total, 7 up, 7 in 2024-05-06T12:29:11.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T12:29:11.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T12:29:11.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:11 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:12 smithi086 ceph-mon[23164]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:12.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:12 smithi067 ceph-mon[27137]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:12.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:12 smithi067 ceph-mon[22413]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:13 smithi086 ceph-mon[23164]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:13 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/4077460982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]: dispatch 2024-05-06T12:29:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:13 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]: dispatch 2024-05-06T12:29:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:13 smithi086 ceph-mon[23164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]': finished 2024-05-06T12:29:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:13 smithi086 ceph-mon[23164]: osdmap e51: 8 total, 7 up, 8 in 2024-05-06T12:29:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:13 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[22413]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/4077460982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]: dispatch 2024-05-06T12:29:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]: dispatch 2024-05-06T12:29:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[22413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]': finished 2024-05-06T12:29:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[22413]: osdmap e51: 8 total, 7 up, 8 in 2024-05-06T12:29:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:13.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[27137]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:13.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/4077460982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]: dispatch 2024-05-06T12:29:13.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]: dispatch 2024-05-06T12:29:13.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[27137]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa7ced44-b5b6-4016-a560-f2a181e64833"}]': finished 2024-05-06T12:29:13.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[27137]: osdmap e51: 8 total, 7 up, 8 in 2024-05-06T12:29:13.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:13 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:14 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/3132047657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:29:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:14 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/3132047657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:29:14.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:14 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/3132047657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T12:29:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:15 smithi086 ceph-mon[23164]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-06T12:29:15.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:15 smithi067 ceph-mon[22413]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-06T12:29:15.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:15 smithi067 ceph-mon[27137]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-06T12:29:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:17 smithi086 ceph-mon[23164]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-06T12:29:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:17 smithi067 ceph-mon[22413]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-06T12:29:17.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:17 smithi067 ceph-mon[27137]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-06T12:29:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:19 smithi086 ceph-mon[23164]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T12:29:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:19 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:29:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:19 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:19 smithi067 ceph-mon[22413]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T12:29:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:19 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:29:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:19 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:19.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:19 smithi067 ceph-mon[27137]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T12:29:19.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:19 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:29:19.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:19 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:20 smithi086 ceph-mon[23164]: Deploying daemon osd.7 on smithi086 2024-05-06T12:29:20.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:20 smithi067 ceph-mon[22413]: Deploying daemon osd.7 on smithi086 2024-05-06T12:29:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:20 smithi067 ceph-mon[27137]: Deploying daemon osd.7 on smithi086 2024-05-06T12:29:21.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:21 smithi086 ceph-mon[23164]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-06T12:29:21.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:21 smithi067 ceph-mon[22413]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-06T12:29:21.876 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:21 smithi067 ceph-mon[27137]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-06T12:29:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:23 smithi086 ceph-mon[23164]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T12:29:23.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:23 smithi067 ceph-mon[22413]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T12:29:23.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:23 smithi067 ceph-mon[27137]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T12:29:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:25 smithi086 ceph-mon[23164]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:25 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:25 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:25.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:25 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:25.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:25 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[22413]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:25.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[27137]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:25 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:26.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:29:26.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:26 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:29:26.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:29:26.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:26 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:29:26.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:29:26.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:26 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:29:27.731 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 7 on host 'smithi086' 2024-05-06T12:29:27.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:27 smithi086 ceph-mon[23164]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:27.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:27 smithi067 ceph-mon[22413]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:27.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:27 smithi067 ceph-mon[27137]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:28.372 DEBUG:teuthology.orchestra.run.smithi086:osd.7> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.7.service 2024-05-06T12:29:28.374 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-05-06T12:29:28.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-06T12:29:28.948 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:28.948 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[22413]: from='osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:29:28.948 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[22413]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:29:28.948 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[22413]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:28.948 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:28.949 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[27137]: from='osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:29:28.949 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[27137]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:29:28.949 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:28 smithi067 ceph-mon[27137]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:29.066 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:28 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:29.066 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:28 smithi086 ceph-mon[23164]: from='osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:29:29.067 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:28 smithi086 ceph-mon[23164]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:29:29.067 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:28 smithi086 ceph-mon[23164]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:29.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:29 smithi086 ceph-mon[23164]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T12:29:29.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:29 smithi086 ceph-mon[23164]: osdmap e52: 8 total, 7 up, 8 in 2024-05-06T12:29:29.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:29 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:29.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:29 smithi086 ceph-mon[23164]: from='osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:29.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:29 smithi086 ceph-mon[23164]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:29.956 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:29 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:29.956 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:29:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[35280]: 2024-05-06T12:29:29.742+0000 7f0eb0fc7700 -1 osd.7 0 waiting for initial osdmap 2024-05-06T12:29:29.956 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:29:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[35280]: 2024-05-06T12:29:29.750+0000 7f0eab15d700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[22413]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[22413]: osdmap e52: 8 total, 7 up, 8 in 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[22413]: from='osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[22413]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[27137]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[27137]: osdmap e52: 8 total, 7 up, 8 in 2024-05-06T12:29:29.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[27137]: from='osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[27137]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:29:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:29 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:29:30.980 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: osdmap e53: 8 total, 7 up, 8 in 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903] boot 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: osdmap e54: 8 total, 8 up, 8 in 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:30.981 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:30 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2645201119' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T12:29:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:29:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: osdmap e53: 8 total, 7 up, 8 in 2024-05-06T12:29:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903] boot 2024-05-06T12:29:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: osdmap e54: 8 total, 8 up, 8 in 2024-05-06T12:29:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2645201119' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: osdmap e53: 8 total, 7 up, 8 in 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: osd.7 [v2:172.21.15.86:6824/151427903,v1:172.21.15.86:6825/151427903] boot 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: osdmap e54: 8 total, 8 up, 8 in 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T12:29:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:30 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2645201119' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T12:29:31.296 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1714998569,"num_in_osds":8,"osd_in_since":1714998553,"num_remapped_pgs":1} 2024-05-06T12:29:31.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T12:29:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:29:31.871 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:31 smithi067 ceph-mon[22413]: purged_snaps scrub starts 2024-05-06T12:29:31.871 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:31 smithi067 ceph-mon[22413]: purged_snaps scrub ok 2024-05-06T12:29:31.871 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:31 smithi067 ceph-mon[22413]: osdmap e55: 8 total, 8 up, 8 in 2024-05-06T12:29:31.871 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:31 smithi067 ceph-mon[27137]: purged_snaps scrub starts 2024-05-06T12:29:31.872 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:31 smithi067 ceph-mon[27137]: purged_snaps scrub ok 2024-05-06T12:29:31.872 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:31 smithi067 ceph-mon[27137]: osdmap e55: 8 total, 8 up, 8 in 2024-05-06T12:29:32.114 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:31 smithi086 ceph-mon[23164]: purged_snaps scrub starts 2024-05-06T12:29:32.114 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:31 smithi086 ceph-mon[23164]: purged_snaps scrub ok 2024-05-06T12:29:32.114 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:31 smithi086 ceph-mon[23164]: osdmap e55: 8 total, 8 up, 8 in 2024-05-06T12:29:33.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: osdmap e56: 8 total, 8 up, 8 in 2024-05-06T12:29:33.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:33.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: Detected new or changed devices on smithi086 2024-05-06T12:29:33.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi086 to 4272M 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: osdmap e56: 8 total, 8 up, 8 in 2024-05-06T12:29:33.001 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: Detected new or changed devices on smithi086 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi086 to 4272M 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.002 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:33.003 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:33.003 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:33.003 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:32 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: osdmap e56: 8 total, 8 up, 8 in 2024-05-06T12:29:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: Detected new or changed devices on smithi086 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi086 to 4272M 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:33.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:32 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:33.385 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:29:33.385 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"fsid":"847b865c-0ba3-11ef-bc97-c7b262605968","created":"2024-05-06T12:24:34.566511+0000","modified":"2024-05-06T12:29:32.963683+0000","last_up_change":"2024-05-06T12:29:29.945683+0000","last_in_change":"2024-05-06T12:29:13.116321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T12:27:34.473704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b8669559-459e-464d-a02a-bc107141d715","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6803","nonce":1842861353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6805","nonce":1842861353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6809","nonce":1842861353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6807","nonce":1842861353}]},"public_addr":"172.21.15.67:6803/1842861353","cluster_addr":"172.21.15.67:6805/1842861353","heartbeat_back_addr":"172.21.15.67:6809/1842861353","heartbeat_front_addr":"172.21.15.67:6807/1842861353","state":["exists","up"]},{"osd":1,"uuid":"88072cfe-96ff-4a30-93db-74402fa65199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6811","nonce":4168781866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6813","nonce":4168781866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6817","nonce":4168781866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6815","nonce":4168781866}]},"public_addr":"172.21.15.67:6811/4168781866","cluster_addr":"172.21.15.67:6813/4168781866","heartbeat_back_addr":"172.21.15.67:6817/4168781866","heartbeat_front_addr":"172.21.15.67:6815/4168781866","state":["exists","up"]},{"osd":2,"uuid":"051681d5-efcb-4a36-81b6-266ffb96dfd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6819","nonce":4011218293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6821","nonce":4011218293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6825","nonce":4011218293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6823","nonce":4011218293}]},"public_addr":"172.21.15.67:6819/4011218293","cluster_addr":"172.21.15.67:6821/4011218293","heartbeat_back_addr":"172.21.15.67:6825/4011218293","heartbeat_front_addr":"172.21.15.67:6823/4011218293","state":["exists","up"]},{"osd":3,"uuid":"315d956c-b263-46c8-97b3-5346b0d05d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6827","nonce":3041731118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6829","nonce":3041731118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6833","nonce":3041731118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6831","nonce":3041731118}]},"public_addr":"172.21.15.67:6827/3041731118","cluster_addr":"172.21.15.67:6829/3041731118","heartbeat_back_addr":"172.21.15.67:6833/3041731118","heartbeat_front_addr":"172.21.15.67:6831/3041731118","state":["exists","up"]},{"osd":4,"uuid":"52e8b1fc-7a3c-4c99-81cd-c690bc506e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6801","nonce":1658830317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6803","nonce":1658830317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6807","nonce":1658830317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6805","nonce":1658830317}]},"public_addr":"172.21.15.86:6801/1658830317","cluster_addr":"172.21.15.86:6803/1658830317","heartbeat_back_addr":"172.21.15.86:6807/1658830317","heartbeat_front_addr":"172.21.15.86:6805/1658830317","state":["exists","up"]},{"osd":5,"uuid":"5df6cce9-b99e-407c-8d88-ed2cd6f136be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6809","nonce":2497673500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6811","nonce":2497673500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6815","nonce":2497673500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6813","nonce":2497673500}]},"public_addr":"172.21.15.86:6809/2497673500","cluster_addr":"172.21.15.86:6811/2497673500","heartbeat_back_addr":"172.21.15.86:6815/2497673500","heartbeat_front_addr":"172.21.15.86:6813/2497673500","state":["exists","up"]},{"osd":6,"uuid":"0b83b846-b7f8-45e9-aecd-1be2a6157b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6817","nonce":3070209570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6819","nonce":3070209570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6823","nonce":3070209570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6821","nonce":3070209570}]},"public_addr":"172.21.15.86:6817/3070209570","cluster_addr":"172.21.15.86:6819/3070209570","heartbeat_back_addr":"172.21.15.86:6823/3070209570","heartbeat_front_addr":"172.21.15.86:6821/3070209570","state":["exists","up"]},{"osd":7,"uuid":"aa7ced44-b5b6-4016-a560-f2a181e64833","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6825","nonce":151427903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6827","nonce":151427903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6831","nonce":151427903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6829","nonce":151427903}]},"public_addr":"172.21.15.86:6825/151427903","cluster_addr":"172.21.15.86:6827/151427903","heartbeat_back_addr":"172.21.15.86:6831/151427903","heartbeat_front_addr":"172.21.15.86:6829/151427903","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:26:43.308084+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:06.836302+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:31.054649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:55.931174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:28:17.553711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:28:41.287102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:29:04.584067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:29:28.858631+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/4180882082":"2024-05-07T12:25:25.912428+0000","172.21.15.67:6800/4180882082":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/388484257":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/2204009303":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/586832651":"2024-05-07T12:25:25.912428+0000","172.21.15.67:6800/1200758237":"2024-05-07T12:25:02.597844+0000","172.21.15.67:6801/1200758237":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/1872361456":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/3823878302":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/1500831993":"2024-05-07T12:25:02.597844+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T12:29:34.030 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T12:27:34.473704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-05-06T12:29:34.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-06T12:29:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:33 smithi067 ceph-mon[27137]: osdmap e57: 8 total, 8 up, 8 in 2024-05-06T12:29:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:33 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/57630008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:29:34.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:33 smithi067 ceph-mon[22413]: osdmap e57: 8 total, 8 up, 8 in 2024-05-06T12:29:34.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:33 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/57630008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:29:34.251 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:29:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:33 smithi086 ceph-mon[23164]: osdmap e57: 8 total, 8 up, 8 in 2024-05-06T12:29:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:33 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/57630008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:29:35.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:34 smithi086 ceph-mon[23164]: osdmap e58: 8 total, 8 up, 8 in 2024-05-06T12:29:35.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:34 smithi086 ceph-mon[23164]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:35.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:34 smithi067 ceph-mon[22413]: osdmap e58: 8 total, 8 up, 8 in 2024-05-06T12:29:35.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:34 smithi067 ceph-mon[22413]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:35.408 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:34 smithi067 ceph-mon[27137]: osdmap e58: 8 total, 8 up, 8 in 2024-05-06T12:29:35.408 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:34 smithi067 ceph-mon[27137]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:36.044 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-06T12:29:36.346 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:36 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2019008971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T12:29:36.347 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:36 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2019008971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T12:29:36.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:36 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2019008971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T12:29:36.773 INFO:tasks.cephadm:Adding prometheus.a on smithi086 2024-05-06T12:29:36.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch apply prometheus '1;smithi086=a' 2024-05-06T12:29:37.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:37 smithi086 ceph-mon[23164]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-06T12:29:37.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:37 smithi067 ceph-mon[22413]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-06T12:29:37.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:37 smithi067 ceph-mon[27137]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-06T12:29:38.711 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled prometheus update... 2024-05-06T12:29:39.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mon[23164]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-06T12:29:39.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:39.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:39.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:39.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:39.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:39.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-06T12:29:39.419 DEBUG:teuthology.orchestra.run.smithi086:prometheus.a> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@prometheus.a.service 2024-05-06T12:29:39.422 INFO:tasks.cephadm:Adding node-exporter.a on smithi067 2024-05-06T12:29:39.422 INFO:tasks.cephadm:Adding node-exporter.b on smithi086 2024-05-06T12:29:39.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch apply node-exporter '2;smithi067=a;smithi086=b' 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[22413]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[27137]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-06T12:29:39.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:39.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:39.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:39.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:39.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' 2024-05-06T12:29:39.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-06T12:29:40.024 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ignoring --setuser ceph since I am not root 2024-05-06T12:29:40.024 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ignoring --setgroup ceph since I am not root 2024-05-06T12:29:40.024 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:39 smithi086 ceph-mgr[24353]: -- 172.21.15.86:0/688240209 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ad35040340 con 0x55ad35018c00 2024-05-06T12:29:40.024 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:39.921+0000 7f0bd3b01000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:29:40.072 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:39 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setuser ceph since I am not root 2024-05-06T12:29:40.072 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:39 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setgroup ceph since I am not root 2024-05-06T12:29:40.072 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:39 smithi067 ceph-mgr[22649]: -- 172.21.15.67:0/2560651084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55781c200340 con 0x55781c1d8800 2024-05-06T12:29:40.072 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:39 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:39.935+0000 7f33f81ec000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:29:40.072 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:40.070+0000 7f33f81ec000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:29:40.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:40.054+0000 7f0bd3b01000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:29:40.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:40.154+0000 7f0bd3b01000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:29:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:40 smithi086 ceph-mon[23164]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:40 smithi086 ceph-mon[23164]: Saving service prometheus spec with placement smithi086=a;count:1 2024-05-06T12:29:40.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:40 smithi086 ceph-mon[23164]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-06T12:29:40.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:40 smithi086 ceph-mon[23164]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-06T12:29:40.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:40.174+0000 7f33f81ec000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:29:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[27137]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[27137]: Saving service prometheus spec with placement smithi086=a;count:1 2024-05-06T12:29:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[27137]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-06T12:29:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[27137]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-06T12:29:40.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[22413]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:40.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[22413]: Saving service prometheus spec with placement smithi086=a;count:1 2024-05-06T12:29:40.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[22413]: from='mgr.14150 172.21.15.67:0/239010133' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-06T12:29:40.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:40 smithi067 ceph-mon[22413]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-06T12:29:40.753 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:40.460+0000 7f0bd3b01000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:29:40.753 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:40.544+0000 7f0bd3b01000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:29:40.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:40.492+0000 7f33f81ec000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:29:40.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:40.578+0000 7f33f81ec000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:29:41.936 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:41 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:41.762+0000 7f33f81ec000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:29:42.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:41.682+0000 7f0bd3b01000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:29:42.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:41.843+0000 7f0bd3b01000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:29:42.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:41.923+0000 7f0bd3b01000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:29:42.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:41 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:41.934+0000 7f33f81ec000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:29:42.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:42.020+0000 7f33f81ec000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:29:42.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:42.228+0000 7f0bd3b01000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:29:42.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:42.315+0000 7f0bd3b01000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:29:42.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:42.405+0000 7f0bd3b01000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:29:42.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:42.347+0000 7f33f81ec000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:29:42.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:42.441+0000 7f33f81ec000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:29:42.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:42.538+0000 7f33f81ec000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:29:43.071 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:42.641+0000 7f0bd3b01000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:29:43.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:42.789+0000 7f33f81ec000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:29:43.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:43.069+0000 7f0bd3b01000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:29:43.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:43.251+0000 7f33f81ec000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:29:43.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:43.546+0000 7f0bd3b01000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:29:43.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:43.636+0000 7f0bd3b01000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:29:44.166 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:43.758+0000 7f33f81ec000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:29:44.166 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:43.856+0000 7f33f81ec000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:29:44.166 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:44.164+0000 7f33f81ec000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:29:44.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:43.922+0000 7f0bd3b01000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:29:44.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:44 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:44.009+0000 7f0bd3b01000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:29:44.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:44.258+0000 7f33f81ec000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:29:44.733 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:44 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:44.403+0000 7f0bd3b01000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:29:44.733 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:44 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:44.532+0000 7f0bd3b01000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:29:44.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:44 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:44.621+0000 7f0bd3b01000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:29:44.734 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:44 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:44.732+0000 7f0bd3b01000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:29:44.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:44.678+0000 7f33f81ec000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:29:44.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:44.817+0000 7f33f81ec000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:29:44.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:44.915+0000 7f33f81ec000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:29:45.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:44 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:44.819+0000 7f0bd3b01000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:29:45.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:45.036+0000 7f33f81ec000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:29:45.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:45.130+0000 7f33f81ec000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:29:45.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:45.361+0000 7f0bd3b01000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:29:45.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:45 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:45.705+0000 7f33f81ec000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:29:45.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[22413]: Standby manager daemon x restarted 2024-05-06T12:29:45.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[22413]: Standby manager daemon x started 2024-05-06T12:29:45.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[27137]: Standby manager daemon x restarted 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[27137]: Standby manager daemon x started 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:29:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:45 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:29:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:45 smithi086 ceph-mon[23164]: Standby manager daemon x restarted 2024-05-06T12:29:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:45 smithi086 ceph-mon[23164]: Standby manager daemon x started 2024-05-06T12:29:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:45 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:29:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:45 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:29:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:45 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:45 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/720244713' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:45.729+0000 7f0bd3b01000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: [06/May/2024:12:29:45] ENGINE Bus STARTING 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: CherryPy Checker: 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: The Application mounted at '' has an empty config. 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: [06/May/2024:12:29:45] ENGINE Serving on http://:::9283 2024-05-06T12:29:46.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:29:45 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: [06/May/2024:12:29:45] ENGINE Bus STARTED 2024-05-06T12:29:46.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:46.086+0000 7f33f81ec000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:29:47.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[22413]: mgrmap e17: y(active, since 4m), standbys: x 2024-05-06T12:29:47.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[22413]: Active manager daemon y restarted 2024-05-06T12:29:47.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[22413]: Activating manager daemon y 2024-05-06T12:29:47.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[22413]: osdmap e59: 8 total, 8 up, 8 in 2024-05-06T12:29:47.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[27137]: mgrmap e17: y(active, since 4m), standbys: x 2024-05-06T12:29:47.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[27137]: Active manager daemon y restarted 2024-05-06T12:29:47.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[27137]: Activating manager daemon y 2024-05-06T12:29:47.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:46 smithi067 ceph-mon[27137]: osdmap e59: 8 total, 8 up, 8 in 2024-05-06T12:29:47.227 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:29:47] ENGINE Bus STARTING 2024-05-06T12:29:47.227 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: CherryPy Checker: 2024-05-06T12:29:47.227 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: The Application mounted at '' has an empty config. 2024-05-06T12:29:47.227 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:29:47.228 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:29:47] ENGINE Serving on http://:::9283 2024-05-06T12:29:47.228 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:29:47] ENGINE Bus STARTED 2024-05-06T12:29:47.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:46 smithi086 ceph-mon[23164]: mgrmap e17: y(active, since 4m), standbys: x 2024-05-06T12:29:47.312 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:46 smithi086 ceph-mon[23164]: Active manager daemon y restarted 2024-05-06T12:29:47.312 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:46 smithi086 ceph-mon[23164]: Activating manager daemon y 2024-05-06T12:29:47.312 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:46 smithi086 ceph-mon[23164]: osdmap e59: 8 total, 8 up, 8 in 2024-05-06T12:29:47.990 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled node-exporter update... 2024-05-06T12:29:48.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: mgrmap e18: y(active, starting, since 0.875293s), standbys: x 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:29:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: Manager daemon y is now available 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:48.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:48.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:48.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:48.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:29:48.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:29:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: mgrmap e18: y(active, starting, since 0.875293s), standbys: x 2024-05-06T12:29:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:29:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:29:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:29:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:29:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:29:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:29:48.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: Manager daemon y is now available 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:29:48.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: mgrmap e18: y(active, starting, since 0.875293s), standbys: x 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:29:48.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: Manager daemon y is now available 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:48.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:29:48.231 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:29:48.231 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:29:48.231 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:48.231 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:29:48.231 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:29:48.937 DEBUG:teuthology.orchestra.run.smithi067:node-exporter.a> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.a.service 2024-05-06T12:29:48.940 DEBUG:teuthology.orchestra.run.smithi086:node-exporter.b> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.b.service 2024-05-06T12:29:48.942 INFO:tasks.cephadm:Adding alertmanager.a on smithi067 2024-05-06T12:29:48.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch apply alertmanager '1;smithi067=a' 2024-05-06T12:29:49.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:48 smithi086 ceph-mon[23164]: mgrmap e19: y(active, since 1.88616s), standbys: x 2024-05-06T12:29:49.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:48 smithi086 ceph-mon[23164]: Saving service node-exporter spec with placement smithi067=a;smithi086=b;count:2 2024-05-06T12:29:49.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:48 smithi086 ceph-mon[23164]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:49.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:48 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:49.164 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:48 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:49.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[22413]: mgrmap e19: y(active, since 1.88616s), standbys: x 2024-05-06T12:29:49.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[22413]: Saving service node-exporter spec with placement smithi067=a;smithi086=b;count:2 2024-05-06T12:29:49.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[22413]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:49.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:49.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:49.171 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:29:48] ENGINE Bus STARTING 2024-05-06T12:29:49.172 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:49 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:29:49] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:29:49.172 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:29:49 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:29:49] ENGINE Bus STARTED 2024-05-06T12:29:49.172 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[27137]: mgrmap e19: y(active, since 1.88616s), standbys: x 2024-05-06T12:29:49.172 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[27137]: Saving service node-exporter spec with placement smithi067=a;smithi086=b;count:2 2024-05-06T12:29:49.172 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[27137]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:49.172 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:49.172 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:48 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.140 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: [06/May/2024:12:29:48] ENGINE Bus STARTING 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: [06/May/2024:12:29:49] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: [06/May/2024:12:29:49] ENGINE Bus STARTED 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.141 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:49 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: [06/May/2024:12:29:48] ENGINE Bus STARTING 2024-05-06T12:29:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-06T12:29:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: [06/May/2024:12:29:49] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:29:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: [06/May/2024:12:29:49] ENGINE Bus STARTED 2024-05-06T12:29:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: [06/May/2024:12:29:48] ENGINE Bus STARTING 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: [06/May/2024:12:29:49] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: [06/May/2024:12:29:49] ENGINE Bus STARTED 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:29:50.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:29:50.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:50.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:49 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.094 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled alertmanager update... 2024-05-06T12:29:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:51 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:29:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:51 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:29:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:51 smithi086 ceph-mon[23164]: Deploying daemon prometheus.a on smithi086 2024-05-06T12:29:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:29:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:29:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[22413]: Deploying daemon prometheus.a on smithi086 2024-05-06T12:29:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:29:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:29:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[27137]: Deploying daemon prometheus.a on smithi086 2024-05-06T12:29:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:52.013 DEBUG:teuthology.orchestra.run.smithi067:alertmanager.a> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@alertmanager.a.service 2024-05-06T12:29:52.015 INFO:tasks.cephadm:Adding grafana.a on smithi086 2024-05-06T12:29:52.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph orch apply grafana '1;smithi086=a' 2024-05-06T12:29:52.315 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:52 smithi086 ceph-mon[23164]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:52.315 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:52 smithi086 ceph-mon[23164]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:52.316 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:52 smithi086 ceph-mon[23164]: Saving service alertmanager spec with placement smithi067=a;count:1 2024-05-06T12:29:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:52 smithi067 ceph-mon[22413]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:52 smithi067 ceph-mon[22413]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:52 smithi067 ceph-mon[22413]: Saving service alertmanager spec with placement smithi067=a;count:1 2024-05-06T12:29:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:52 smithi067 ceph-mon[27137]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:52 smithi067 ceph-mon[27137]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:52 smithi067 ceph-mon[27137]: Saving service alertmanager spec with placement smithi067=a;count:1 2024-05-06T12:29:54.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:54 smithi086 ceph-mon[23164]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:54 smithi067 ceph-mon[22413]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:54.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:54 smithi067 ceph-mon[27137]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:54.767 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled grafana update... 2024-05-06T12:29:55.725 DEBUG:teuthology.orchestra.run.smithi086:grafana.a> sudo journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@grafana.a.service 2024-05-06T12:29:55.727 INFO:tasks.cephadm:Setting up client nodes... 2024-05-06T12:29:55.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-06T12:29:56.050 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:55 smithi067 ceph-mon[27137]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:56.050 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:55 smithi067 ceph-mon[27137]: Saving service grafana spec with placement smithi086=a;count:1 2024-05-06T12:29:56.050 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:55 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:56.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:55 smithi067 ceph-mon[22413]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:56.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:55 smithi067 ceph-mon[22413]: Saving service grafana spec with placement smithi086=a;count:1 2024-05-06T12:29:56.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:55 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:55 smithi086 ceph-mon[23164]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:29:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:55 smithi086 ceph-mon[23164]: Saving service grafana spec with placement smithi086=a;count:1 2024-05-06T12:29:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:55 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:29:57.059 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:56 smithi067 ceph-mon[22413]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:57.060 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:56 smithi067 ceph-mon[27137]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:57.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:56 smithi086 ceph-mon[23164]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:57.973 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-05-06T12:29:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBFzThm7xK0ORAAgOGcVEtZQwOU1k7jJnNEGQ== 2024-05-06T12:29:58.656 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-06T12:29:58.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-06T12:29:58.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-06T12:29:58.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-06T12:29:58.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:58 smithi067 ceph-mon[22413]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:58.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:58 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2379057812' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:29:58.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:29:58 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2379057812' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T12:29:58.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:58 smithi067 ceph-mon[27137]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:58.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:58 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2379057812' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:29:58.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:29:58 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2379057812' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T12:29:59.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:58 smithi086 ceph-mon[23164]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:29:59.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:58 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2379057812' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:29:59.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:29:58 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2379057812' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T12:30:00.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:00 smithi086 ceph-mon[23164]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:00.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:00 smithi086 ceph-mon[23164]: overall HEALTH_OK 2024-05-06T12:30:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:00 smithi067 ceph-mon[22413]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:01.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:00 smithi067 ceph-mon[22413]: overall HEALTH_OK 2024-05-06T12:30:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:00 smithi067 ceph-mon[27137]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:00 smithi067 ceph-mon[27137]: overall HEALTH_OK 2024-05-06T12:30:02.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:02 smithi067 ceph-mon[22413]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:02.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:02 smithi067 ceph-mon[27137]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:03.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:02 smithi086 ceph-mon[23164]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:03.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:02 smithi086 systemd[1]: Starting Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:30:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-05-06T12:30:04.035 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBMzThmCv++ARAAitlXwydXopU7LgXLg62D2g== 2024-05-06T12:30:04.142 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:03 smithi086 podman[38204]: 2024-05-06 12:30:03.783111297 +0000 UTC m=+0.021067957 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T12:30:04.142 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:03 smithi086 podman[38204]: 2024-05-06 12:30:03.88028461 +0000 UTC m=+0.118241259 container create 9886608b24c3a4f476e0d3050838409a4a10a42bf83588b95e7e5790f3c05c9c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:30:04.142 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 podman[38204]: 2024-05-06 12:30:04.141391366 +0000 UTC m=+0.379348013 container init 9886608b24c3a4f476e0d3050838409a4a10a42bf83588b95e7e5790f3c05c9c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 podman[38204]: 2024-05-06 12:30:04.144090347 +0000 UTC m=+0.382046998 container start 9886608b24c3a4f476e0d3050838409a4a10a42bf83588b95e7e5790f3c05c9c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.167Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.167Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.167Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.167Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi086 (none))" 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.167Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.167Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.168Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T12:30:04.620 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.168Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.169Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.170Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.170Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.619µs 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.170Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.171Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.171Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.548µs wal_replay_duration=269.407µs total_replay_duration=303.184µs 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.171Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.171Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.171Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.187Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.072634ms db_storage=610ns remote_storage=1.616µs web_handler=269ns query_engine=648ns scrape=417.295µs scrape_sd=15.324µs notify=859ns notify_sd=2.384µs rules=14.475049ms 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:30:04.187Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 bash[38204]: 9886608b24c3a4f476e0d3050838409a4a10a42bf83588b95e7e5790f3c05c9c 2024-05-06T12:30:04.621 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:30:04 smithi086 systemd[1]: Started Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:30:04.878 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:04 smithi086 ceph-mon[23164]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:04.879 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:04 smithi086 ceph-mon[23164]: from='client.? 172.21.15.86:0/1838626596' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:30:04.879 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:04 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:30:04.879 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:04 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T12:30:04.879 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:04 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:04.901 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-06T12:30:04.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-06T12:30:04.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-06T12:30:04.937 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-06T12:30:04.937 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-06T12:30:04.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[22413]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[22413]: from='client.? 172.21.15.86:0/1838626596' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[27137]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[27137]: from='client.? 172.21.15.86:0/1838626596' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:30:04.941 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T12:30:04.942 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T12:30:04.942 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:04 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:05.225 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:05 smithi067 bash[41801]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-05-06T12:30:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:05 smithi067 ceph-mon[22413]: Deploying daemon node-exporter.a on smithi067 2024-05-06T12:30:05.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:05 smithi067 ceph-mon[27137]: Deploying daemon node-exporter.a on smithi067 2024-05-06T12:30:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:05 smithi086 ceph-mon[23164]: Deploying daemon node-exporter.a on smithi067 2024-05-06T12:30:06.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:06 smithi067 ceph-mon[22413]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:06.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:06 smithi067 ceph-mon[27137]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:07.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:06 smithi086 ceph-mon[23164]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:07.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:30:07.824 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"active_gid":24343,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":976842880},{"type":"v1","addr":"172.21.15.67:6801","nonce":976842880}]},"active_addr":"172.21.15.67:6801/976842880","active_change":"2024-05-06T12:29:46.089824+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24356,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.67:8443/","prometheus":"http://172.21.15.67:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2089669879}]},{"addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2116552457}]},{"addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3470008396}]},{"addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":289594580}]}]}} 2024-05-06T12:30:07.829 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-06T12:30:07.829 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-06T12:30:07.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T12:30:07.975 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:07 smithi067 bash[41801]: Getting image source signatures 2024-05-06T12:30:07.975 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:07 smithi067 bash[41801]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-05-06T12:30:07.975 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:07 smithi067 bash[41801]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-05-06T12:30:07.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:07 smithi067 bash[41801]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-05-06T12:30:08.059 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:08.278 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:08 smithi067 ceph-mon[27137]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:08.279 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:08 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:08.279 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:08 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/125938856' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T12:30:08.279 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:08 smithi067 ceph-mon[22413]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:08.279 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:08 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:08.279 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:08 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/125938856' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T12:30:08.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:08 smithi086 ceph-mon[23164]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:08.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:08 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:08.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:08 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/125938856' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T12:30:09.946 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:09 smithi067 bash[41801]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-05-06T12:30:10.225 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:09 smithi067 bash[41801]: Writing manifest to image destination 2024-05-06T12:30:10.225 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 podman[41801]: 2024-05-06 12:30:10.184949357 +0000 UTC m=+5.128207524 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T12:30:10.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-mon[22413]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:10.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:10 smithi067 ceph-mon[27137]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:10.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:10 smithi086 ceph-mon[23164]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:10.774 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:10] "GET /metrics HTTP/1.1" 200 191512 "" "Prometheus/2.33.4" 2024-05-06T12:30:10.774 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 podman[41801]: 2024-05-06 12:30:10.612707202 +0000 UTC m=+5.555965354 container create 50bfd17cec6591f6f4f7ca4af80e14c6533107f260346c025bc38369314cc2c2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:30:11.060 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 podman[41801]: 2024-05-06 12:30:10.96731659 +0000 UTC m=+5.910574744 container init 50bfd17cec6591f6f4f7ca4af80e14c6533107f260346c025bc38369314cc2c2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:30:11.060 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 podman[41801]: 2024-05-06 12:30:10.970406995 +0000 UTC m=+5.913665149 container start 50bfd17cec6591f6f4f7ca4af80e14c6533107f260346c025bc38369314cc2c2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:30:11.060 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.974Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T12:30:11.060 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.974Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T12:30:11.060 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.974Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T12:30:11.060 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.974Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T12:30:11.060 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T12:30:11.061 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.975Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T12:30:11.062 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T12:30:11.063 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[42083]: ts=2024-05-06T12:30:10.976Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T12:30:11.064 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:11 smithi067 bash[41801]: 50bfd17cec6591f6f4f7ca4af80e14c6533107f260346c025bc38369314cc2c2 2024-05-06T12:30:11.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:30:11 smithi067 systemd[1]: Started Ceph node-exporter.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:30:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:30:11.475 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"fsid":"847b865c-0ba3-11ef-bc97-c7b262605968","created":"2024-05-06T12:24:34.566511+0000","modified":"2024-05-06T12:29:46.089412+0000","last_up_change":"2024-05-06T12:29:29.945683+0000","last_in_change":"2024-05-06T12:29:13.116321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T12:27:34.473704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b8669559-459e-464d-a02a-bc107141d715","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6803","nonce":1842861353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6805","nonce":1842861353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6809","nonce":1842861353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6807","nonce":1842861353}]},"public_addr":"172.21.15.67:6803/1842861353","cluster_addr":"172.21.15.67:6805/1842861353","heartbeat_back_addr":"172.21.15.67:6809/1842861353","heartbeat_front_addr":"172.21.15.67:6807/1842861353","state":["exists","up"]},{"osd":1,"uuid":"88072cfe-96ff-4a30-93db-74402fa65199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6811","nonce":4168781866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6813","nonce":4168781866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6817","nonce":4168781866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6815","nonce":4168781866}]},"public_addr":"172.21.15.67:6811/4168781866","cluster_addr":"172.21.15.67:6813/4168781866","heartbeat_back_addr":"172.21.15.67:6817/4168781866","heartbeat_front_addr":"172.21.15.67:6815/4168781866","state":["exists","up"]},{"osd":2,"uuid":"051681d5-efcb-4a36-81b6-266ffb96dfd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6819","nonce":4011218293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6821","nonce":4011218293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6825","nonce":4011218293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6823","nonce":4011218293}]},"public_addr":"172.21.15.67:6819/4011218293","cluster_addr":"172.21.15.67:6821/4011218293","heartbeat_back_addr":"172.21.15.67:6825/4011218293","heartbeat_front_addr":"172.21.15.67:6823/4011218293","state":["exists","up"]},{"osd":3,"uuid":"315d956c-b263-46c8-97b3-5346b0d05d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6827","nonce":3041731118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6829","nonce":3041731118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6833","nonce":3041731118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6831","nonce":3041731118}]},"public_addr":"172.21.15.67:6827/3041731118","cluster_addr":"172.21.15.67:6829/3041731118","heartbeat_back_addr":"172.21.15.67:6833/3041731118","heartbeat_front_addr":"172.21.15.67:6831/3041731118","state":["exists","up"]},{"osd":4,"uuid":"52e8b1fc-7a3c-4c99-81cd-c690bc506e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6801","nonce":1658830317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6803","nonce":1658830317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6807","nonce":1658830317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6805","nonce":1658830317}]},"public_addr":"172.21.15.86:6801/1658830317","cluster_addr":"172.21.15.86:6803/1658830317","heartbeat_back_addr":"172.21.15.86:6807/1658830317","heartbeat_front_addr":"172.21.15.86:6805/1658830317","state":["exists","up"]},{"osd":5,"uuid":"5df6cce9-b99e-407c-8d88-ed2cd6f136be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6809","nonce":2497673500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6811","nonce":2497673500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6815","nonce":2497673500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6813","nonce":2497673500}]},"public_addr":"172.21.15.86:6809/2497673500","cluster_addr":"172.21.15.86:6811/2497673500","heartbeat_back_addr":"172.21.15.86:6815/2497673500","heartbeat_front_addr":"172.21.15.86:6813/2497673500","state":["exists","up"]},{"osd":6,"uuid":"0b83b846-b7f8-45e9-aecd-1be2a6157b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6817","nonce":3070209570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6819","nonce":3070209570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6823","nonce":3070209570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6821","nonce":3070209570}]},"public_addr":"172.21.15.86:6817/3070209570","cluster_addr":"172.21.15.86:6819/3070209570","heartbeat_back_addr":"172.21.15.86:6823/3070209570","heartbeat_front_addr":"172.21.15.86:6821/3070209570","state":["exists","up"]},{"osd":7,"uuid":"aa7ced44-b5b6-4016-a560-f2a181e64833","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6825","nonce":151427903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6827","nonce":151427903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6831","nonce":151427903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6829","nonce":151427903}]},"public_addr":"172.21.15.86:6825/151427903","cluster_addr":"172.21.15.86:6827/151427903","heartbeat_back_addr":"172.21.15.86:6831/151427903","heartbeat_front_addr":"172.21.15.86:6829/151427903","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:26:43.308084+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:06.836302+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:31.054649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:55.931174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:28:17.553711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:28:41.287102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:29:04.584067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:29:28.858631+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2422689785":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6800/1588658531":"2024-05-07T12:29:46.089361+0000","172.21.15.67:0/4062172122":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6801/4180882082":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/2195830361":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6800/4180882082":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/388484257":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/712625767":"2024-05-07T12:29:46.089361+0000","172.21.15.67:0/2204009303":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/586832651":"2024-05-07T12:25:25.912428+0000","172.21.15.67:6800/1200758237":"2024-05-07T12:25:02.597844+0000","172.21.15.67:6801/1588658531":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6801/1200758237":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/1872361456":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/3823878302":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/1500831993":"2024-05-07T12:25:02.597844+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T12:30:11.783 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:30:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:30:12.119 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:11 smithi086 bash[38509]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-05-06T12:30:12.164 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-06T12:30:12.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T12:30:12.385 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[22413]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[22413]: Deploying daemon node-exporter.b on smithi086 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2962645059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[27137]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[27137]: Deploying daemon node-exporter.b on smithi086 2024-05-06T12:30:12.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:12 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2962645059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:30:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:12 smithi086 ceph-mon[23164]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:12 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:12 smithi086 ceph-mon[23164]: Deploying daemon node-exporter.b on smithi086 2024-05-06T12:30:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:12 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2962645059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:30:12.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:12 smithi086 bash[38509]: Getting image source signatures 2024-05-06T12:30:12.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:12 smithi086 bash[38509]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-05-06T12:30:12.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:12 smithi086 bash[38509]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-05-06T12:30:12.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:12 smithi086 bash[38509]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-05-06T12:30:14.119 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:13 smithi086 bash[38509]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-05-06T12:30:14.119 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:13 smithi086 bash[38509]: Writing manifest to image destination 2024-05-06T12:30:14.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:30:14.268 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"fsid":"847b865c-0ba3-11ef-bc97-c7b262605968","created":"2024-05-06T12:24:34.566511+0000","modified":"2024-05-06T12:29:46.089412+0000","last_up_change":"2024-05-06T12:29:29.945683+0000","last_in_change":"2024-05-06T12:29:13.116321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T12:27:34.473704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b8669559-459e-464d-a02a-bc107141d715","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6803","nonce":1842861353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6805","nonce":1842861353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6809","nonce":1842861353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1842861353},{"type":"v1","addr":"172.21.15.67:6807","nonce":1842861353}]},"public_addr":"172.21.15.67:6803/1842861353","cluster_addr":"172.21.15.67:6805/1842861353","heartbeat_back_addr":"172.21.15.67:6809/1842861353","heartbeat_front_addr":"172.21.15.67:6807/1842861353","state":["exists","up"]},{"osd":1,"uuid":"88072cfe-96ff-4a30-93db-74402fa65199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6811","nonce":4168781866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6813","nonce":4168781866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6817","nonce":4168781866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":4168781866},{"type":"v1","addr":"172.21.15.67:6815","nonce":4168781866}]},"public_addr":"172.21.15.67:6811/4168781866","cluster_addr":"172.21.15.67:6813/4168781866","heartbeat_back_addr":"172.21.15.67:6817/4168781866","heartbeat_front_addr":"172.21.15.67:6815/4168781866","state":["exists","up"]},{"osd":2,"uuid":"051681d5-efcb-4a36-81b6-266ffb96dfd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6819","nonce":4011218293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6821","nonce":4011218293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6825","nonce":4011218293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4011218293},{"type":"v1","addr":"172.21.15.67:6823","nonce":4011218293}]},"public_addr":"172.21.15.67:6819/4011218293","cluster_addr":"172.21.15.67:6821/4011218293","heartbeat_back_addr":"172.21.15.67:6825/4011218293","heartbeat_front_addr":"172.21.15.67:6823/4011218293","state":["exists","up"]},{"osd":3,"uuid":"315d956c-b263-46c8-97b3-5346b0d05d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6827","nonce":3041731118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6829","nonce":3041731118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6833","nonce":3041731118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3041731118},{"type":"v1","addr":"172.21.15.67:6831","nonce":3041731118}]},"public_addr":"172.21.15.67:6827/3041731118","cluster_addr":"172.21.15.67:6829/3041731118","heartbeat_back_addr":"172.21.15.67:6833/3041731118","heartbeat_front_addr":"172.21.15.67:6831/3041731118","state":["exists","up"]},{"osd":4,"uuid":"52e8b1fc-7a3c-4c99-81cd-c690bc506e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6801","nonce":1658830317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6803","nonce":1658830317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6807","nonce":1658830317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1658830317},{"type":"v1","addr":"172.21.15.86:6805","nonce":1658830317}]},"public_addr":"172.21.15.86:6801/1658830317","cluster_addr":"172.21.15.86:6803/1658830317","heartbeat_back_addr":"172.21.15.86:6807/1658830317","heartbeat_front_addr":"172.21.15.86:6805/1658830317","state":["exists","up"]},{"osd":5,"uuid":"5df6cce9-b99e-407c-8d88-ed2cd6f136be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6809","nonce":2497673500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6811","nonce":2497673500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6815","nonce":2497673500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2497673500},{"type":"v1","addr":"172.21.15.86:6813","nonce":2497673500}]},"public_addr":"172.21.15.86:6809/2497673500","cluster_addr":"172.21.15.86:6811/2497673500","heartbeat_back_addr":"172.21.15.86:6815/2497673500","heartbeat_front_addr":"172.21.15.86:6813/2497673500","state":["exists","up"]},{"osd":6,"uuid":"0b83b846-b7f8-45e9-aecd-1be2a6157b54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6817","nonce":3070209570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6819","nonce":3070209570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6823","nonce":3070209570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3070209570},{"type":"v1","addr":"172.21.15.86:6821","nonce":3070209570}]},"public_addr":"172.21.15.86:6817/3070209570","cluster_addr":"172.21.15.86:6819/3070209570","heartbeat_back_addr":"172.21.15.86:6823/3070209570","heartbeat_front_addr":"172.21.15.86:6821/3070209570","state":["exists","up"]},{"osd":7,"uuid":"aa7ced44-b5b6-4016-a560-f2a181e64833","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6825","nonce":151427903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6827","nonce":151427903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6831","nonce":151427903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":151427903},{"type":"v1","addr":"172.21.15.86:6829","nonce":151427903}]},"public_addr":"172.21.15.86:6825/151427903","cluster_addr":"172.21.15.86:6827/151427903","heartbeat_back_addr":"172.21.15.86:6831/151427903","heartbeat_front_addr":"172.21.15.86:6829/151427903","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:26:43.308084+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:06.836302+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:31.054649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:27:55.931174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:28:17.553711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:28:41.287102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:29:04.584067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T12:29:28.858631+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2422689785":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6800/1588658531":"2024-05-07T12:29:46.089361+0000","172.21.15.67:0/4062172122":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6801/4180882082":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/2195830361":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6800/4180882082":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/388484257":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/712625767":"2024-05-07T12:29:46.089361+0000","172.21.15.67:0/2204009303":"2024-05-07T12:25:25.912428+0000","172.21.15.67:0/586832651":"2024-05-07T12:25:25.912428+0000","172.21.15.67:6800/1200758237":"2024-05-07T12:25:02.597844+0000","172.21.15.67:6801/1588658531":"2024-05-07T12:29:46.089361+0000","172.21.15.67:6801/1200758237":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/1872361456":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/3823878302":"2024-05-07T12:25:02.597844+0000","172.21.15.67:0/1500831993":"2024-05-07T12:25:02.597844+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T12:30:14.268 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:14 smithi067 ceph-mon[27137]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:14.268 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:14 smithi067 ceph-mon[22413]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:14.418 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-mon[23164]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:14.418 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 podman[38509]: 2024-05-06 12:30:14.125034908 +0000 UTC m=+2.242281586 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T12:30:14.418 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 podman[38509]: 2024-05-06 12:30:14.232015848 +0000 UTC m=+2.349262510 container create b88ef5b0017c7ac6e8a65a9284395727a88ecc99fb35aa9eabda4a7e6b77cff9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:30:14.418 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 podman[38509]: 2024-05-06 12:30:14.417424263 +0000 UTC m=+2.534670925 container init b88ef5b0017c7ac6e8a65a9284395727a88ecc99fb35aa9eabda4a7e6b77cff9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:30:14.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-06T12:30:14.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-06T12:30:14.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-06T12:30:14.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-06T12:30:14.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-06T12:30:14.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-06T12:30:14.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-06T12:30:14.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 podman[38509]: 2024-05-06 12:30:14.420121532 +0000 UTC m=+2.537368197 container start b88ef5b0017c7ac6e8a65a9284395727a88ecc99fb35aa9eabda4a7e6b77cff9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.423Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.423Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.424Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.424Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T12:30:14.870 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T12:30:14.871 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T12:30:14.872 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T12:30:14.873 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T12:30:14.874 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[38560]: ts=2024-05-06T12:30:14.425Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T12:30:14.874 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 bash[38509]: b88ef5b0017c7ac6e8a65a9284395727a88ecc99fb35aa9eabda4a7e6b77cff9 2024-05-06T12:30:14.874 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:30:14 smithi086 systemd[1]: Started Ceph node-exporter.b for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:30:15.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3795469752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:30:15.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:15.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:30:15.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:30:15.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:30:15.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3795469752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:30:15.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:15.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:30:15.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:30:15.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:15 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:30:15.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:15 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3795469752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T12:30:15.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:15 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:15.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:15 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:30:15.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:15 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:30:15.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:15 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:30:15.548 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:15.549 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:15.883 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:16.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:16 smithi067 ceph-mon[22413]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:16.280 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:16 smithi067 ceph-mon[27137]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:16.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:16 smithi086 ceph-mon[23164]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:18.336 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:18 smithi067 ceph-mon[22413]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:18.336 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:18 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:18.336 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:18 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:18.337 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:18 smithi067 ceph-mon[27137]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:18.337 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:18 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:18.337 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:18 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:18.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:18 smithi086 ceph-mon[23164]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:18.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:18 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:18.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:18 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:20.367 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:20 smithi067 ceph-mon[22413]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:20.368 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:20 smithi067 ceph-mon[27137]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:20.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:20 smithi086 ceph-mon[23164]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:20.703 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:30:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:20] "GET /metrics HTTP/1.1" 200 191489 "" "Prometheus/2.33.4" 2024-05-06T12:30:21.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:30:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:30:22.291 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:22 smithi067 ceph-mon[22413]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:22.292 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:22 smithi067 ceph-mon[27137]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:22.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:22 smithi086 ceph-mon[23164]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:22.380 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:23.375 INFO:teuthology.orchestra.run.smithi067.stdout:128849018905 2024-05-06T12:30:23.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-06T12:30:24.247 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:24.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:24 smithi067 ceph-mon[22413]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:24.301 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:24 smithi067 ceph-mon[27137]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:24.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:24 smithi086 ceph-mon[23164]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:24.966 INFO:teuthology.orchestra.run.smithi067.stdout:231928233995 2024-05-06T12:30:24.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-06T12:30:25.534 INFO:teuthology.orchestra.run.smithi067.stdout:163208757271 2024-05-06T12:30:25.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-06T12:30:26.126 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:25 smithi067 ceph-mon[22413]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:26.126 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:25 smithi067 ceph-mon[27137]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:26.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:25 smithi086 ceph-mon[23164]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:26.840 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[27137]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[27137]: Deploying daemon alertmanager.a on smithi067 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[22413]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:28 smithi067 ceph-mon[22413]: Deploying daemon alertmanager.a on smithi067 2024-05-06T12:30:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:28 smithi086 ceph-mon[23164]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:28 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:28 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:28 smithi086 ceph-mon[23164]: Deploying daemon alertmanager.a on smithi067 2024-05-06T12:30:29.146 INFO:teuthology.orchestra.run.smithi067.stdout:34359738415 2024-05-06T12:30:29.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-06T12:30:30.364 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:30 smithi067 ceph-mon[22413]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:30.365 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:30 smithi067 ceph-mon[27137]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:30.389 INFO:teuthology.orchestra.run.smithi067.stdout:55834574890 2024-05-06T12:30:30.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-06T12:30:30.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:30 smithi086 ceph-mon[23164]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:30.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:30:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:30] "GET /metrics HTTP/1.1" 200 191489 "" "Prometheus/2.33.4" 2024-05-06T12:30:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:31.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:30:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:30:32.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:32 smithi086 ceph-mon[23164]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:32 smithi067 ceph-mon[22413]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:32 smithi067 ceph-mon[27137]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:33.099 INFO:teuthology.orchestra.run.smithi067.stdout:197568495635 2024-05-06T12:30:33.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-06T12:30:33.841 INFO:teuthology.orchestra.run.smithi067.stdout:107374182433 2024-05-06T12:30:33.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-06T12:30:34.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:34 smithi086 ceph-mon[23164]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:34.657 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:34 smithi067 ceph-mon[27137]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:34.657 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:34 smithi067 ceph-mon[22413]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:35.328 INFO:teuthology.orchestra.run.smithi067.stdout:77309411365 2024-05-06T12:30:35.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-06T12:30:35.948 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:35.949 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:35.949 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:36.217 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:35 smithi067 ceph-mon[22413]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:36.217 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:35 smithi067 ceph-mon[27137]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:36.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:35 smithi086 ceph-mon[23164]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:38.251 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:38.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:38 smithi067 ceph-mon[22413]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:38.358 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:38 smithi067 ceph-mon[27137]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:38 smithi086 ceph-mon[23164]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:39.725 INFO:teuthology.orchestra.run.smithi067.stdout:231928233998 2024-05-06T12:30:40.308 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:40 smithi086 ceph-mon[23164]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:40.308 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:40 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3558233151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-06T12:30:40.351 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:40 smithi067 ceph-mon[22413]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:40.351 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:40 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3558233151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-06T12:30:40.351 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:40 smithi067 ceph-mon[27137]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:40.351 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:40 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3558233151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-06T12:30:40.639 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:30:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:40] "GET /metrics HTTP/1.1" 200 191487 "" "Prometheus/2.33.4" 2024-05-06T12:30:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2491912776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T12:30:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:41 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2491912776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T12:30:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:41 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2491912776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T12:30:41.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:30:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:30:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:42 smithi067 ceph-mon[22413]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:42.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:42 smithi067 ceph-mon[27137]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:42 smithi086 ceph-mon[23164]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:43.366 INFO:teuthology.orchestra.run.smithi067.stdout:128849018909 2024-05-06T12:30:44.218 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:44 smithi067 ceph-mon[27137]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:44.218 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:44 smithi067 ceph-mon[22413]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:44 smithi086 ceph-mon[23164]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:44.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233995 got 231928233998 for osd.7 2024-05-06T12:30:44.759 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:44.842 INFO:teuthology.orchestra.run.smithi067.stdout:34359738418 2024-05-06T12:30:45.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:45 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/477441321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T12:30:45.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:45 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/477441321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T12:30:45.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:45 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/477441321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T12:30:46.336 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:46 smithi067 ceph-mon[22413]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:46.336 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:46 smithi067 ceph-mon[27137]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:46 smithi086 ceph-mon[23164]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:47.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:30:47.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:30:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:30:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:30:47.416 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:30:47.416 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:30:47.616 INFO:teuthology.orchestra.run.smithi067.stdout:163208757275 2024-05-06T12:30:47.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738415 got 34359738418 for osd.0 2024-05-06T12:30:47.828 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:48 smithi086 ceph-mon[23164]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:48 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1177034265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T12:30:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:48 smithi067 ceph-mon[27137]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:48 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1177034265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T12:30:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:48 smithi067 ceph-mon[22413]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:48 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1177034265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T12:30:48.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757275 for osd.5 2024-05-06T12:30:48.633 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:49.044 INFO:teuthology.orchestra.run.smithi067.stdout:55834574893 2024-05-06T12:30:49.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:49 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3175041527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T12:30:49.390 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:49 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3175041527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T12:30:49.390 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:49 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3175041527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T12:30:49.811 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574893 for osd.1 2024-05-06T12:30:49.811 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:50.314 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018905 got 128849018909 for osd.4 2024-05-06T12:30:50.314 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:50 smithi086 ceph-mon[23164]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:50.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:50 smithi067 ceph-mon[22413]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:50.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:50 smithi067 ceph-mon[27137]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:50.688 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:30:50 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:50] "GET /metrics HTTP/1.1" 200 191500 "" "Prometheus/2.33.4" 2024-05-06T12:30:51.595 INFO:teuthology.orchestra.run.smithi067.stdout:77309411369 2024-05-06T12:30:51.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:30:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:30:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:30:52.148 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:52 smithi067 ceph-mon[22413]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:52.148 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:52 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2484271085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T12:30:52.148 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:52 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2384601376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T12:30:52.149 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:52 smithi067 ceph-mon[27137]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:52.149 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:52 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2484271085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T12:30:52.149 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:52 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2384601376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T12:30:52.199 INFO:teuthology.orchestra.run.smithi067.stdout:197568495639 2024-05-06T12:30:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:52 smithi086 ceph-mon[23164]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:52 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2484271085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T12:30:52.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:52 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2384601376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T12:30:53.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:53 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3822074514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-06T12:30:53.386 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:53 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3822074514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-06T12:30:53.386 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:53 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3822074514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-06T12:30:53.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411369 for osd.2 2024-05-06T12:30:53.620 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:53.674 INFO:teuthology.orchestra.run.smithi067.stdout:107374182437 2024-05-06T12:30:54.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:54 smithi067 ceph-mon[22413]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:54.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:54 smithi067 ceph-mon[27137]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:54.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182433 got 107374182437 for osd.3 2024-05-06T12:30:54.266 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:54.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:54 smithi086 ceph-mon[23164]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:54.843 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495635 got 197568495639 for osd.6 2024-05-06T12:30:54.843 DEBUG:teuthology.parallel:result is None 2024-05-06T12:30:54.843 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-06T12:30:54.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-06T12:30:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:56.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:55 smithi067 ceph-mon[27137]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:55 smithi067 ceph-mon[22413]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:56.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:55 smithi086 ceph-mon[23164]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:57.620 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:30:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-06T12:30:58.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:58 smithi067 ceph-mon[27137]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:58 smithi067 ceph-mon[27137]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T12:30:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:58 smithi067 ceph-mon[22413]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:58 smithi067 ceph-mon[22413]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T12:30:58.227 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:30:58 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[45310]: level=info ts=2024-05-06T12:30:58.102Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-06T12:30:58.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:58 smithi086 ceph-mon[23164]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:30:58.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:58 smithi086 ceph-mon[23164]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T12:30:58.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:30:58 smithi067 bash[45283]: 0bfc21926e7db4c018df603a67307f842efec7fae2951166fe1dba36ee1131a2 2024-05-06T12:30:58.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:30:58 smithi067 systemd[1]: Started Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:30:58.853 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-05-06T12:30:56.998092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48848,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912368,"statfs":{"total":767960285184,"available":767910264832,"internally_reserved":0,"allocated":6242304,"data_stored":3488924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002201"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-06T12:29:47.063002+0000","last_change":"2024-05-06T12:29:33.975772+0000","last_active":"2024-05-06T12:29:47.063002+0000","last_peered":"2024-05-06T12:29:47.063002+0000","last_clean":"2024-05-06T12:29:47.063002+0000","last_became_active":"2024-05-06T12:29:33.975241+0000","last_became_peered":"2024-05-06T12:29:33.975241+0000","last_unstale":"2024-05-06T12:29:47.063002+0000","last_undegraded":"2024-05-06T12:29:47.063002+0000","last_fullsized":"2024-05-06T12:29:47.063002+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T12:27:34.474136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T12:27:34.474136+0000","last_clean_scrub_stamp":"2024-05-06T12:27:34.474136+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T23:46:10.041100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.74299999999999999}]},{"osd":2,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.80100000000000005}]},{"osd":3,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.63600000000000001}]},{"osd":4,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.498},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.51900000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.64700000000000002}]},{"osd":1,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.747}]},{"osd":2,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.67000000000000004}]},{"osd":4,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon May 6 12:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.33800000000000002},"min":{"1min":0.27700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65100000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.38,"15min":0.34999999999999998},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.58499999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon May 6 12:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"min":{"1min":0.27700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.624,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.44500000000000001,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.223}]},{"osd":3,"last update":"Mon May 6 12:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27100000000000002}]},{"osd":4,"last update":"Mon May 6 12:30:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81599999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon May 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65000000000000002,"15min":0.65700000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63200000000000001,"15min":0.63500000000000001},"min":{"1min":0.41999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51400000000000001}]},{"osd":6,"last update":"Mon May 6 12:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.621},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon May 6 12:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.72699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 6 12:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.371,"15min":0.37},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52000000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.36899999999999999,"15min":0.34799999999999998},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon May 6 12:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42199999999999999,"15min":0.41199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40200000000000002,"15min":0.39000000000000001},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.51400000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.39000000000000001}]},{"osd":3,"last update":"Mon May 6 12:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon May 6 12:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon May 6 12:30:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.41099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.60399999999999998}]},{"osd":6,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon May 6 12:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.65200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40000000000000002,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41099999999999998,"15min":0.40799999999999997},"min":{"1min":0.26500000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.375}]},{"osd":1,"last update":"Mon May 6 12:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40100000000000002,"15min":0.40799999999999997},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57599999999999996,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.27900000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66300000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.497,"15min":0.496},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46300000000000002,"15min":0.46000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.0720000000000001,"15min":1.1479999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79500000000000004,"5min":12.887,"15min":12.887},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon May 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.129,"15min":1.2230000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71699999999999997,"5min":12.920999999999999,"15min":12.920999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon May 6 12:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon May 6 12:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55700000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.375,"15min":0.375},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.5},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.434,"15min":0.42999999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.44500000000000001}]},{"osd":1,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44,"15min":0.438},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59199999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.623,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.45500000000000002,"15min":0.45900000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45900000000000002}]},{"osd":4,"last update":"Mon May 6 12:30:22 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon May 6 12:30:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.628,"15min":0.63},"min":{"1min":0.44800000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78700000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.622},"min":{"1min":0.437,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77800000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.69999999999999996}]},{"osd":6,"last update":"Mon May 6 12:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.75600000000000001}]},{"osd":7,"last update":"Mon May 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.746}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.5,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.53500000000000003}]},{"osd":1,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.34899999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.62,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69699999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.45800000000000002}]},{"osd":2,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.641,"15min":1.827},"min":{"1min":0.38500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":18.736999999999998,"15min":18.736999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.63,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":1.641,"15min":1.829},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.70199999999999996,"5min":18.725000000000001,"15min":18.725000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.29799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.34799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.35599999999999998}]},{"osd":6,"last update":"Mon May 6 12:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.44},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon May 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.44700000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53000000000000003,"15min":0.53800000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69799999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.502},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65300000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.498}]},{"osd":1,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50700000000000001,"15min":0.503},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67100000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.495,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.747,"15min":0.747},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64900000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.39700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41199999999999998,"15min":0.40799999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.59699999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.307,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63500000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon May 6 12:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T12:30:58.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-06T12:30:59.079 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:30:59.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[22413]: Deploying daemon grafana.a on smithi086 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:30:59 smithi067 ceph-mon[27137]: Deploying daemon grafana.a on smithi086 2024-05-06T12:30:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:30:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:59 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:30:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:30:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:30:59 smithi086 ceph-mon[23164]: Deploying daemon grafana.a on smithi086 2024-05-06T12:31:00.177 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[45310]: level=info ts=2024-05-06T12:31:00.074Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00011726s 2024-05-06T12:31:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:00 smithi067 ceph-mon[22413]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:00.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:00 smithi067 ceph-mon[27137]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:00.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:31:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:00] "GET /metrics HTTP/1.1" 200 191500 "" "Prometheus/2.33.4" 2024-05-06T12:31:00.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:31:00.865 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-06T12:31:00.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:00 smithi086 ceph-mon[23164]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:01.512 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-05-06T12:30:58.998432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48848,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912368,"statfs":{"total":767960285184,"available":767910264832,"internally_reserved":0,"allocated":6242304,"data_stored":3488924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002155"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-06T12:29:47.063002+0000","last_change":"2024-05-06T12:29:33.975772+0000","last_active":"2024-05-06T12:29:47.063002+0000","last_peered":"2024-05-06T12:29:47.063002+0000","last_clean":"2024-05-06T12:29:47.063002+0000","last_became_active":"2024-05-06T12:29:33.975241+0000","last_became_peered":"2024-05-06T12:29:33.975241+0000","last_unstale":"2024-05-06T12:29:47.063002+0000","last_undegraded":"2024-05-06T12:29:47.063002+0000","last_fullsized":"2024-05-06T12:29:47.063002+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T12:27:34.474136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T12:27:34.474136+0000","last_clean_scrub_stamp":"2024-05-06T12:27:34.474136+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T23:46:10.041100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.56200000000000006}]},{"osd":1,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.502},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.65200000000000002}]},{"osd":4,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.627}]},{"osd":5,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.66600000000000004}]},{"osd":6,"last update":"Mon May 6 12:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53900000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45000000000000001}]},{"osd":1,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54000000000000004}]},{"osd":3,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.498},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon May 6 12:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.58699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.33800000000000002},"min":{"1min":0.27700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65100000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.38,"15min":0.34999999999999998},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.58499999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon May 6 12:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"min":{"1min":0.27700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.624,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.44500000000000001,"15min":0.433},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.223}]},{"osd":3,"last update":"Mon May 6 12:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27100000000000002}]},{"osd":4,"last update":"Mon May 6 12:30:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81599999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon May 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65000000000000002,"15min":0.65700000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63200000000000001,"15min":0.63500000000000001},"min":{"1min":0.41999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51400000000000001}]},{"osd":6,"last update":"Mon May 6 12:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.621},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon May 6 12:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.72699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 6 12:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.371,"15min":0.37},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52000000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.36899999999999999,"15min":0.34799999999999998},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.252}]},{"osd":2,"last update":"Mon May 6 12:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42199999999999999,"15min":0.41199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40200000000000002,"15min":0.39000000000000001},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.51400000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.28299999999999997}]},{"osd":3,"last update":"Mon May 6 12:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.26900000000000002}]},{"osd":4,"last update":"Mon May 6 12:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon May 6 12:30:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.41099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.45500000000000002}]},{"osd":6,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon May 6 12:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56299999999999994}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40000000000000002,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41099999999999998,"15min":0.40799999999999997},"min":{"1min":0.26500000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.59099999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon May 6 12:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40100000000000002,"15min":0.40799999999999997},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57599999999999996,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.27900000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66300000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.375}]},{"osd":3,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.497,"15min":0.496},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.434},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46300000000000002,"15min":0.46000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.0720000000000001,"15min":1.1479999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79500000000000004,"5min":12.887,"15min":12.887},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon May 6 12:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.129,"15min":1.2230000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71699999999999997,"5min":12.920999999999999,"15min":12.920999999999999},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon May 6 12:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon May 6 12:30:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.499},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57099999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.375,"15min":0.375},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.434,"15min":0.42999999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.67200000000000004}]},{"osd":1,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44,"15min":0.438},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59199999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.623,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon May 6 12:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.746},{"interface":"front","average":{"1min":0.434,"5min":0.45500000000000002,"15min":0.45900000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.63100000000000001}]},{"osd":4,"last update":"Mon May 6 12:30:22 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.70999999999999996}]},{"osd":5,"last update":"Mon May 6 12:30:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.628,"15min":0.63},"min":{"1min":0.44800000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78700000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.62,"15min":0.622},"min":{"1min":0.437,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77800000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.82499999999999996}]},{"osd":6,"last update":"Mon May 6 12:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon May 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.80000000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.5,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.34899999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.62,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69699999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.641,"15min":1.827},"min":{"1min":0.38500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":18.736999999999998,"15min":18.736999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon May 6 12:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.63,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":1.641,"15min":1.829},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.70199999999999996,"5min":18.725000000000001,"15min":18.725000000000001},"last":0.38300000000000001}]},{"osd":5,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.29799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.34799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon May 6 12:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.624}]},{"osd":7,"last update":"Mon May 6 12:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.56599999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53000000000000003,"15min":0.53800000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69799999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.502},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65300000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.57399999999999995}]},{"osd":1,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50700000000000001,"15min":0.503},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67100000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.495,"5min":0.504,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.747,"15min":0.747},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64900000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.39700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.746}]},{"osd":4,"last update":"Mon May 6 12:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41199999999999998,"15min":0.40799999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.59699999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.307,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63500000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon May 6 12:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon May 6 12:30:33 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T12:31:01.513 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-06T12:31:01.514 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-06T12:31:01.514 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-06T12:31:01.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-06T12:31:01.736 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T12:31:01.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:31:01 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:31:02.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:02 smithi067 ceph-mon[22413]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T12:31:02.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:02 smithi067 ceph-mon[22413]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:02.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:02.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:02 smithi067 ceph-mon[27137]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T12:31:02.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:02 smithi067 ceph-mon[27137]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:02.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:02.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:02 smithi086 ceph-mon[23164]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T12:31:02.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:02 smithi086 ceph-mon[23164]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:02.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-06T12:31:03.540 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-06T12:31:04.150 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-06T12:31:04.150 INFO:tasks.cephadm:Setup complete, yielding 2024-05-06T12:31:04.150 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T12:31:04.160 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-06T12:31:04.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-05-06T12:31:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:04 smithi067 ceph-mon[27137]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:04 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2532139971' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T12:31:04.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:04 smithi067 ceph-mon[22413]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:04.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:04 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2532139971' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T12:31:04.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:04 smithi086 ceph-mon[23164]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:04.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:04 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2532139971' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T12:31:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:05 smithi086 ceph-mon[23164]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:06.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:05 smithi067 ceph-mon[22413]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:06.452 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:05 smithi067 ceph-mon[27137]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:06.752 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T12:31:06.762 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-06T12:31:06.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-05-06T12:31:07.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:07 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' 2024-05-06T12:31:07.326 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:07 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' 2024-05-06T12:31:07.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:07 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' 2024-05-06T12:31:08.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:08 smithi067 ceph-mon[22413]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:08.290 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:08 smithi067 ceph-mon[27137]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:08.290 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:08 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[45310]: level=info ts=2024-05-06T12:31:08.076Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002272921s 2024-05-06T12:31:08.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:08 smithi086 ceph-mon[23164]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:10.205 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:31:10.205 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "6aadaceb-6a22-4199-832d-f9f7ffa2a4b3", 2024-05-06T12:31:10.205 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "r", 2024-05-06T12:31:10.205 INFO:teuthology.orchestra.run.smithi067.stdout: "current_period": "37e3cf85-c5a7-4103-91cc-344243b83a73", 2024-05-06T12:31:10.205 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 1 2024-05-06T12:31:10.206 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:31:10.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:10 smithi067 ceph-mon[22413]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:10.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:10 smithi067 ceph-mon[22413]: osdmap e60: 8 total, 8 up, 8 in 2024-05-06T12:31:10.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:10 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1438819349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-06T12:31:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:10 smithi067 ceph-mon[27137]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:10 smithi067 ceph-mon[27137]: osdmap e60: 8 total, 8 up, 8 in 2024-05-06T12:31:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:10 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1438819349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-06T12:31:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:10 smithi086 ceph-mon[23164]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:10 smithi086 ceph-mon[23164]: osdmap e60: 8 total, 8 up, 8 in 2024-05-06T12:31:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:10 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1438819349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-06T12:31:10.775 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:31:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:10] "GET /metrics HTTP/1.1" 200 191496 "" "Prometheus/2.33.4" 2024-05-06T12:31:10.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-05-06T12:31:11.417 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:11 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1438819349' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-06T12:31:11.417 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:11 smithi086 ceph-mon[23164]: osdmap e61: 8 total, 8 up, 8 in 2024-05-06T12:31:11.417 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:11 smithi086 ceph-mon[23164]: osdmap e62: 8 total, 8 up, 8 in 2024-05-06T12:31:11.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:11 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1438819349' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-06T12:31:11.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:11 smithi067 ceph-mon[22413]: osdmap e61: 8 total, 8 up, 8 in 2024-05-06T12:31:11.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:11 smithi067 ceph-mon[22413]: osdmap e62: 8 total, 8 up, 8 in 2024-05-06T12:31:11.454 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:11 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1438819349' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-06T12:31:11.454 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:11 smithi067 ceph-mon[27137]: osdmap e61: 8 total, 8 up, 8 in 2024-05-06T12:31:11.454 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:11 smithi067 ceph-mon[27137]: osdmap e62: 8 total, 8 up, 8 in 2024-05-06T12:31:11.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:31:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:31:12.443 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:12 smithi067 ceph-mon[22413]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:12.443 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:12 smithi067 ceph-mon[27137]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:12.502 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:31:12.502 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "50e144b8-fc8a-45a3-9c6a-6975391b8b8f", 2024-05-06T12:31:12.502 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default", 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "api_name": "default", 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "is_master": "true", 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "endpoints": [], 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "hostnames": [], 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "hostnames_s3website": [], 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "master_zone": "", 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "zones": [], 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_targets": [], 2024-05-06T12:31:12.503 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2024-05-06T12:31:12.504 INFO:teuthology.orchestra.run.smithi067.stdout: "realm_id": "6aadaceb-6a22-4199-832d-f9f7ffa2a4b3", 2024-05-06T12:31:12.504 INFO:teuthology.orchestra.run.smithi067.stdout: "sync_policy": { 2024-05-06T12:31:12.504 INFO:teuthology.orchestra.run.smithi067.stdout: "groups": [] 2024-05-06T12:31:12.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:12.504 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:31:12.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:12 smithi086 ceph-mon[23164]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:13.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-05-06T12:31:14.119 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 podman[39240]: 2024-05-06 12:31:13.91438208 +0000 UTC m=+0.017737742 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-05-06T12:31:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:14 smithi067 ceph-mon[22413]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-05-06T12:31:14.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:14 smithi067 ceph-mon[27137]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-05-06T12:31:14.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:14 smithi086 ceph-mon[23164]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-05-06T12:31:14.910 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 podman[39240]: 2024-05-06 12:31:14.657658347 +0000 UTC m=+0.761014038 container create 87f103efe219c8478872744f5445261cc3fc7d1ebeb0c5172cf1b6a3a8f92a7d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, version=8.5, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-05-06T12:31:14.910 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 podman[39240]: 2024-05-06 12:31:14.859653729 +0000 UTC m=+0.963009380 container init 87f103efe219c8478872744f5445261cc3fc7d1ebeb0c5172cf1b6a3a8f92a7d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.component=ubi8-container, name=ubi8, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, version=8.5, io.openshift.expose-services=, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8) 2024-05-06T12:31:14.910 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 podman[39240]: 2024-05-06 12:31:14.863107747 +0000 UTC m=+0.966463396 container start 87f103efe219c8478872744f5445261cc3fc7d1ebeb0c5172cf1b6a3a8f92a7d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, version=8.5, com.redhat.component=ubi8-container, architecture=x86_64, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , distribution-scope=public, vcs-type=git, io.openshift.expose-services=, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.buildah.version=1.24.2) 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-05-06T12:31:14.911 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="App mode production" logger=settings 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-05-06T12:31:14.912 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-05-06T12:31:15.163 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-05-06T12:31:15.164 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 bash[39240]: 87f103efe219c8478872744f5445261cc3fc7d1ebeb0c5172cf1b6a3a8f92a7d 2024-05-06T12:31:15.165 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-05-06T12:31:15.165 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-05-06T12:31:15.165 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 systemd[1]: Started Ceph grafana.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:31:15.165 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-05-06T12:31:15.165 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-05-06T12:31:15.165 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-05-06T12:31:15.165 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-05-06T12:31:15.166 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-05-06T12:31:15.167 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:14+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-05-06T12:31:15.168 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-05-06T12:31:15.169 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-05-06T12:31:15.170 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-05-06T12:31:15.170 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-05-06T12:31:15.170 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-05-06T12:31:15.170 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-05-06T12:31:15.170 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-05-06T12:31:15.170 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-05-06T12:31:15.170 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-05-06T12:31:15.171 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-05-06T12:31:15.172 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-05-06T12:31:15.173 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-05-06T12:31:15.174 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-05-06T12:31:15.175 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-05-06T12:31:15.176 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-05-06T12:31:15.176 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "0c175ef4-4d5f-48e9-9dd3-9fae2f9e4705", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "z", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "domain_root": "z.rgw.meta:root", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "control_pool": "z.rgw.control", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "gc_pool": "z.rgw.log:gc", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "lc_pool": "z.rgw.log:lc", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "log_pool": "z.rgw.log", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "roles_pool": "z.rgw.meta:roles", 2024-05-06T12:31:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "otp_pool": "z.rgw.otp", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "system_key": { 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "access_key": "", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "" 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_pools": [ 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "default-placement", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "val": { 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "index_pool": "z.rgw.buckets.index", 2024-05-06T12:31:15.178 INFO:teuthology.orchestra.run.smithi067.stdout: "storage_classes": { 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: "STANDARD": { 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: "data_pool": "z.rgw.buckets.data" 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: "index_type": 0 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: "realm_id": "6aadaceb-6a22-4199-832d-f9f7ffa2a4b3", 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout: "notif_pool": "z.rgw.log:notif" 2024-05-06T12:31:15.179 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:31:15.414 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-05-06T12:31:15.414 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-05-06T12:31:15.414 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-05-06T12:31:15.415 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-05-06T12:31:15.416 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-05-06T12:31:15.416 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-05-06T12:31:15.416 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-05-06T12:31:15.417 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-05-06T12:31:15.418 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-05-06T12:31:15.419 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-05-06T12:31:15.420 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-05-06T12:31:15.421 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-05-06T12:31:15.422 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-05-06T12:31:15.423 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-05-06T12:31:15.425 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-05-06T12:31:15.425 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-05-06T12:31:15.425 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-05-06T12:31:15.425 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-05-06T12:31:15.425 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-05-06T12:31:15.425 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-05-06T12:31:15.425 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-05-06T12:31:15.426 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-05-06T12:31:15.427 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-05-06T12:31:15.427 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-05-06T12:31:15.427 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-05-06T12:31:15.427 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-05-06T12:31:15.427 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-05-06T12:31:15.427 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-05-06T12:31:15.428 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-05-06T12:31:15.429 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-05-06T12:31:15.430 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-05-06T12:31:15.430 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-05-06T12:31:15.430 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-05-06T12:31:15.430 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-05-06T12:31:15.676 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-05-06T12:31:15.676 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-05-06T12:31:15.676 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-05-06T12:31:15.676 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-05-06T12:31:15.676 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-05-06T12:31:15.676 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-05-06T12:31:15.677 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-05-06T12:31:15.678 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=562.354428ms 2024-05-06T12:31:15.679 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-05-06T12:31:15.680 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:31:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:31:15+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-05-06T12:31:15.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-05-06T12:31:16.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[22413]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-05-06T12:31:16.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:16.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:16.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:16.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[27137]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-05-06T12:31:16.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:16.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:16.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:16 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:16.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:16 smithi086 ceph-mon[23164]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-05-06T12:31:16.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:16 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:16.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:16 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:16.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:16 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:18.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:18 smithi067 ceph-mon[22413]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 522 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-05-06T12:31:18.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:18 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:18.183 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:18 smithi067 ceph-mon[27137]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 522 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-05-06T12:31:18.183 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:18 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:18.365 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:18 smithi086 ceph-mon[23164]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 522 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-05-06T12:31:18.366 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:18 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:19.702 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:19 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:19.702 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:19 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:19 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:20.581 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[22413]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-06T12:31:20.581 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[22413]: osdmap e63: 8 total, 8 up, 8 in 2024-05-06T12:31:20.581 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2459196014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T12:31:20.581 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T12:31:20.581 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:20.581 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:20.582 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[27137]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-06T12:31:20.582 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[27137]: osdmap e63: 8 total, 8 up, 8 in 2024-05-06T12:31:20.582 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2459196014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T12:31:20.582 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T12:31:20.582 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:20.582 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:20 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:20 smithi086 ceph-mon[23164]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 811 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-06T12:31:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:20 smithi086 ceph-mon[23164]: osdmap e63: 8 total, 8 up, 8 in 2024-05-06T12:31:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:20 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2459196014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T12:31:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:20 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T12:31:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:20 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:20 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:20.970 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:31:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:20] "GET /metrics HTTP/1.1" 200 192938 "" "Prometheus/2.33.4" 2024-05-06T12:31:21.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[27137]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T12:31:21.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[27137]: Reconfiguring daemon alertmanager.a on smithi067 2024-05-06T12:31:21.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-06T12:31:21.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[27137]: osdmap e64: 8 total, 8 up, 8 in 2024-05-06T12:31:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[22413]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T12:31:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[22413]: Reconfiguring daemon alertmanager.a on smithi067 2024-05-06T12:31:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-06T12:31:21.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:21 smithi067 ceph-mon[22413]: osdmap e64: 8 total, 8 up, 8 in 2024-05-06T12:31:21.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:31:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:31:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:21 smithi086 ceph-mon[23164]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T12:31:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:21 smithi086 ceph-mon[23164]: Reconfiguring daemon alertmanager.a on smithi067 2024-05-06T12:31:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:21 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-06T12:31:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:21 smithi086 ceph-mon[23164]: osdmap e64: 8 total, 8 up, 8 in 2024-05-06T12:31:22.635 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[27137]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 639 B/s wr, 1 op/s 2024-05-06T12:31:22.635 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[27137]: osdmap e65: 8 total, 8 up, 8 in 2024-05-06T12:31:22.635 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2459196014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T12:31:22.636 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T12:31:22.636 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:22.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[22413]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 639 B/s wr, 1 op/s 2024-05-06T12:31:22.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[22413]: osdmap e65: 8 total, 8 up, 8 in 2024-05-06T12:31:22.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2459196014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T12:31:22.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T12:31:22.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:22 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:22.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:22 smithi086 ceph-mon[23164]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 639 B/s wr, 1 op/s 2024-05-06T12:31:22.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:22 smithi086 ceph-mon[23164]: osdmap e65: 8 total, 8 up, 8 in 2024-05-06T12:31:22.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:22 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2459196014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T12:31:22.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:22 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T12:31:22.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:22 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:23.670 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:23 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-06T12:31:23.671 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:23 smithi067 ceph-mon[22413]: osdmap e66: 8 total, 8 up, 8 in 2024-05-06T12:31:23.671 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:23 smithi067 systemd[1]: Stopping Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:31:23.671 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:23 smithi067 bash[47016]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T12:31:23.671 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:23 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-06T12:31:23.671 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:23 smithi067 ceph-mon[27137]: osdmap e66: 8 total, 8 up, 8 in 2024-05-06T12:31:23.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:23 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-06T12:31:23.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:23 smithi086 ceph-mon[23164]: osdmap e66: 8 total, 8 up, 8 in 2024-05-06T12:31:23.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[45310]: level=info ts=2024-05-06T12:31:23.749Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-06T12:31:23.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:23 smithi067 podman[47027]: 2024-05-06 12:31:23.770252137 +0000 UTC m=+0.149385578 container died 0bfc21926e7db4c018df603a67307f842efec7fae2951166fe1dba36ee1131a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:24.423 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 podman[47027]: 2024-05-06 12:31:24.257881716 +0000 UTC m=+0.637015149 container cleanup 0bfc21926e7db4c018df603a67307f842efec7fae2951166fe1dba36ee1131a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:24.424 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 bash[47027]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a 2024-05-06T12:31:24.424 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 bash[47061]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T12:31:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[22413]: pgmap v58: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-06T12:31:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[22413]: osdmap e67: 8 total, 8 up, 8 in 2024-05-06T12:31:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T12:31:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T12:31:24.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 podman[47042]: 2024-05-06 12:31:24.534270136 +0000 UTC m=+0.759367907 container remove 0bfc21926e7db4c018df603a67307f842efec7fae2951166fe1dba36ee1131a2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:24.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 podman[47042]: 2024-05-06 12:31:24.593177137 +0000 UTC m=+0.818274910 volume remove f717aec538faab7e11dda3bc903f6404d6e18fcc304186187af3bbc5be775176 2024-05-06T12:31:24.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-06T12:31:24.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 systemd[1]: Stopped Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:31:24.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 systemd[1]: Starting Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:31:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[27137]: pgmap v58: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-06T12:31:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[27137]: osdmap e67: 8 total, 8 up, 8 in 2024-05-06T12:31:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T12:31:24.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:24 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T12:31:24.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:24 smithi086 ceph-mon[23164]: pgmap v58: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-06T12:31:24.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:24 smithi086 ceph-mon[23164]: osdmap e67: 8 total, 8 up, 8 in 2024-05-06T12:31:24.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:24 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T12:31:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:24 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T12:31:25.151 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 podman[47134]: 2024-05-06 12:31:24.724110564 +0000 UTC m=+0.018772989 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-05-06T12:31:25.151 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 podman[47134]: 2024-05-06 12:31:24.827268349 +0000 UTC m=+0.121930775 volume create bf1bff2f461a7ac04e571e84dddb318bcf5ef8f7df0381d93a3c62f6f69d9b2c 2024-05-06T12:31:25.152 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:24 smithi067 podman[47134]: 2024-05-06 12:31:24.936190978 +0000 UTC m=+0.230853403 container create f43d96904837b660b5b06d2a9111ac8534d05a2445c2b8b83a4d28b44ebe9df1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:25.708 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:25 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-06T12:31:25.708 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:25 smithi086 ceph-mon[23164]: osdmap e68: 8 total, 8 up, 8 in 2024-05-06T12:31:25.708 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:25 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T12:31:25.708 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:25 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T12:31:25.708 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:25 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[22413]: osdmap e68: 8 total, 8 up, 8 in 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[27137]: osdmap e68: 8 total, 8 up, 8 in 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T12:31:25.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:25 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 podman[47134]: 2024-05-06 12:31:25.23910716 +0000 UTC m=+0.533769586 container init f43d96904837b660b5b06d2a9111ac8534d05a2445c2b8b83a4d28b44ebe9df1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 podman[47134]: 2024-05-06 12:31:25.241906891 +0000 UTC m=+0.536569317 container start f43d96904837b660b5b06d2a9111ac8534d05a2445c2b8b83a4d28b44ebe9df1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.252Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.252Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.253Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.67 port=9094 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.255Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.281Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.281Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.283Z caller=main.go:518 msg=Listening address=:9093 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:25.283Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 bash[47134]: f43d96904837b660b5b06d2a9111ac8534d05a2445c2b8b83a4d28b44ebe9df1 2024-05-06T12:31:25.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:25 smithi067 systemd[1]: Started Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:31:26.493 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:31:26.493 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "936845fe-0389-4836-96f1-edacff90bbc9", 2024-05-06T12:31:26.493 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 1, 2024-05-06T12:31:26.494 INFO:teuthology.orchestra.run.smithi067.stdout: "predecessor_uuid": "37e3cf85-c5a7-4103-91cc-344243b83a73", 2024-05-06T12:31:26.494 INFO:teuthology.orchestra.run.smithi067.stdout: "sync_status": [], 2024-05-06T12:31:26.494 INFO:teuthology.orchestra.run.smithi067.stdout: "period_map": { 2024-05-06T12:31:26.494 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "936845fe-0389-4836-96f1-edacff90bbc9", 2024-05-06T12:31:26.494 INFO:teuthology.orchestra.run.smithi067.stdout: "zonegroups": [ 2024-05-06T12:31:26.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-06T12:31:26.495 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "50e144b8-fc8a-45a3-9c6a-6975391b8b8f", 2024-05-06T12:31:26.495 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default", 2024-05-06T12:31:26.495 INFO:teuthology.orchestra.run.smithi067.stdout: "api_name": "default", 2024-05-06T12:31:26.495 INFO:teuthology.orchestra.run.smithi067.stdout: "is_master": "true", 2024-05-06T12:31:26.495 INFO:teuthology.orchestra.run.smithi067.stdout: "endpoints": [], 2024-05-06T12:31:26.496 INFO:teuthology.orchestra.run.smithi067.stdout: "hostnames": [], 2024-05-06T12:31:26.496 INFO:teuthology.orchestra.run.smithi067.stdout: "hostnames_s3website": [], 2024-05-06T12:31:26.496 INFO:teuthology.orchestra.run.smithi067.stdout: "master_zone": "0c175ef4-4d5f-48e9-9dd3-9fae2f9e4705", 2024-05-06T12:31:26.496 INFO:teuthology.orchestra.run.smithi067.stdout: "zones": [ 2024-05-06T12:31:26.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-06T12:31:26.496 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "0c175ef4-4d5f-48e9-9dd3-9fae2f9e4705", 2024-05-06T12:31:26.496 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "z", 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "endpoints": [], 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "log_meta": "false", 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "log_data": "false", 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_index_max_shards": 11, 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "read_only": "false", 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_type": "", 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "sync_from_all": "true", 2024-05-06T12:31:26.497 INFO:teuthology.orchestra.run.smithi067.stdout: "sync_from": [], 2024-05-06T12:31:26.498 INFO:teuthology.orchestra.run.smithi067.stdout: "redirect_zone": "" 2024-05-06T12:31:26.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:26.498 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:31:26.498 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_targets": [ 2024-05-06T12:31:26.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-06T12:31:26.498 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default-placement", 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: "storage_classes": [ 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: "STANDARD" 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "default-placement", 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: "realm_id": "6aadaceb-6a22-4199-832d-f9f7ffa2a4b3", 2024-05-06T12:31:26.499 INFO:teuthology.orchestra.run.smithi067.stdout: "sync_policy": { 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: "groups": [] 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: "short_zone_ids": [ 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "0c175ef4-4d5f-48e9-9dd3-9fae2f9e4705", 2024-05-06T12:31:26.500 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 4041151413 2024-05-06T12:31:26.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:26.501 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-06T12:31:26.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:26.501 INFO:teuthology.orchestra.run.smithi067.stdout: "master_zonegroup": "50e144b8-fc8a-45a3-9c6a-6975391b8b8f", 2024-05-06T12:31:26.501 INFO:teuthology.orchestra.run.smithi067.stdout: "master_zone": "0c175ef4-4d5f-48e9-9dd3-9fae2f9e4705", 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: "period_config": { 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2024-05-06T12:31:26.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:26.503 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2024-05-06T12:31:26.503 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2024-05-06T12:31:26.504 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2024-05-06T12:31:26.504 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2024-05-06T12:31:26.504 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2024-05-06T12:31:26.504 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: "user_ratelimit": { 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: "max_read_ops": 0, 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: "max_write_ops": 0, 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: "max_read_bytes": 0, 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: "max_write_bytes": 0, 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false 2024-05-06T12:31:26.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_ratelimit": { 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "max_read_ops": 0, 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "max_write_ops": 0, 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "max_read_bytes": 0, 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "max_write_bytes": 0, 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "anonymous_ratelimit": { 2024-05-06T12:31:26.506 INFO:teuthology.orchestra.run.smithi067.stdout: "max_read_ops": 0, 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: "max_write_ops": 0, 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: "max_read_bytes": 0, 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: "max_write_bytes": 0, 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: "realm_id": "6aadaceb-6a22-4199-832d-f9f7ffa2a4b3", 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: "realm_name": "r", 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout: "realm_epoch": 2 2024-05-06T12:31:26.507 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[22413]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[22413]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[22413]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[22413]: osdmap e69: 8 total, 8 up, 8 in 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[27137]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-06T12:31:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[27137]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:31:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[27137]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:31:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-06T12:31:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[27137]: osdmap e69: 8 total, 8 up, 8 in 2024-05-06T12:31:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T12:31:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:26 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T12:31:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:26 smithi086 ceph-mon[23164]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-06T12:31:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:26 smithi086 ceph-mon[23164]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:31:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:26 smithi086 ceph-mon[23164]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:31:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:26 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-06T12:31:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:26 smithi086 ceph-mon[23164]: osdmap e69: 8 total, 8 up, 8 in 2024-05-06T12:31:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:26 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3947444385' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T12:31:26.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:26 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T12:31:27.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-05-06T12:31:27.505 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:27 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-06T12:31:27.505 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:27 smithi086 ceph-mon[23164]: osdmap e70: 8 total, 8 up, 8 in 2024-05-06T12:31:27.508 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:27.256Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000927806s 2024-05-06T12:31:27.508 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:27 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-06T12:31:27.508 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:27 smithi067 ceph-mon[22413]: osdmap e70: 8 total, 8 up, 8 in 2024-05-06T12:31:27.509 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:27 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-06T12:31:27.509 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:27 smithi067 ceph-mon[27137]: osdmap e70: 8 total, 8 up, 8 in 2024-05-06T12:31:28.448 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 systemd[1]: Stopping Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:31:28.448 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 bash[39934]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T12:31:28.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:28 smithi067 ceph-mon[22413]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:28.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:28 smithi067 ceph-mon[27137]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:28.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:28 smithi086 ceph-mon[23164]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.453Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T12:31:28.763 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T12:31:28.764 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T12:31:28.764 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T12:31:28.764 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T12:31:28.764 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[38237]: ts=2024-05-06T12:31:28.454Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T12:31:28.764 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 podman[39945]: 2024-05-06 12:31:28.463909449 +0000 UTC m=+0.135851445 container died 9886608b24c3a4f476e0d3050838409a4a10a42bf83588b95e7e5790f3c05c9c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:29.096 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 podman[39945]: 2024-05-06 12:31:28.861484358 +0000 UTC m=+0.533426344 container cleanup 9886608b24c3a4f476e0d3050838409a4a10a42bf83588b95e7e5790f3c05c9c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:29.097 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 bash[39945]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T12:31:29.097 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:28 smithi086 bash[39971]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T12:31:29.369 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 podman[39959]: 2024-05-06 12:31:29.095951016 +0000 UTC m=+0.634592928 container remove 9886608b24c3a4f476e0d3050838409a4a10a42bf83588b95e7e5790f3c05c9c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:29.369 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T12:31:29.369 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 systemd[1]: Stopped Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:31:29.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 systemd[1]: Starting Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:31:29.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 podman[40040]: 2024-05-06 12:31:29.313673263 +0000 UTC m=+0.096609027 container create af867ca274deb443ff21d77543dfa1aa536f13f87938130ef2a0516496b2705c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:29.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 podman[40040]: 2024-05-06 12:31:29.236180628 +0000 UTC m=+0.019116385 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T12:31:29.499 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled rgw.foo update... 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 podman[40040]: 2024-05-06 12:31:29.490916399 +0000 UTC m=+0.273852155 container init af867ca274deb443ff21d77543dfa1aa536f13f87938130ef2a0516496b2705c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 podman[40040]: 2024-05-06 12:31:29.493378741 +0000 UTC m=+0.276314507 container start af867ca274deb443ff21d77543dfa1aa536f13f87938130ef2a0516496b2705c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.517Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.517Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.517Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.517Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi086 (none))" 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.517Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T12:31:29.692 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.517Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.518Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.519Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.521Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.522Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.522Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.509µs 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.522Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 bash[40040]: af867ca274deb443ff21d77543dfa1aa536f13f87938130ef2a0516496b2705c 2024-05-06T12:31:29.693 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 systemd[1]: Started Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:31:30.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.731Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-05-06T12:31:30.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.733Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-05-06T12:31:30.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.733Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.741µs wal_replay_duration=211.065727ms total_replay_duration=211.103945ms 2024-05-06T12:31:30.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.735Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T12:31:30.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.735Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T12:31:30.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.735Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T12:31:30.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.749Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.291479ms db_storage=696ns remote_storage=1.747µs web_handler=310ns query_engine=741ns scrape=358.435µs scrape_sd=24.282µs notify=19.068µs notify_sd=10.563µs rules=13.655252ms 2024-05-06T12:31:30.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:31:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:31:29.749Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T12:31:30.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch apply rgw smpl' 2024-05-06T12:31:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-05-06T12:31:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: Saving service rgw.foo spec with placement count:2 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.67:9093"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.67:9093"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:31:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:31:30.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-05-06T12:31:30.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-05-06T12:31:30.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:30.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:30 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:30.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-05-06T12:31:30.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:30.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: Saving service rgw.foo spec with placement count:2 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.67:9093"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.67:9093"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:31:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: Saving service rgw.foo spec with placement count:2 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:31:30.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.67:9093"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.67:9093"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.86:9095"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.86:3000"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:30.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:30 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:32 smithi067 ceph-mon[22413]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-06T12:31:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:32 smithi067 ceph-mon[27137]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-06T12:31:32.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:32 smithi086 ceph-mon[23164]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-06T12:31:33.104 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled rgw.smpl update... 2024-05-06T12:31:33.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph osd pool create foo' 2024-05-06T12:31:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[27137]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-06T12:31:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[27137]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[27137]: Saving service rgw.smpl spec with placement count:2 2024-05-06T12:31:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:34.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[22413]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-06T12:31:34.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:34.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[22413]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:34.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[22413]: Saving service rgw.smpl spec with placement count:2 2024-05-06T12:31:34.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:34 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:34 smithi086 ceph-mon[23164]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-06T12:31:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:34 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:34 smithi086 ceph-mon[23164]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:34 smithi086 ceph-mon[23164]: Saving service rgw.smpl spec with placement count:2 2024-05-06T12:31:34.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:34 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:35.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:31:35 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:31:35.257Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001973536s 2024-05-06T12:31:36.164 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:35 smithi067 ceph-mon[22413]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 748 B/s wr, 1 op/s 2024-05-06T12:31:36.164 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:35 smithi067 ceph-mon[27137]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 748 B/s wr, 1 op/s 2024-05-06T12:31:36.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:35 smithi086 ceph-mon[23164]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 748 B/s wr, 1 op/s 2024-05-06T12:31:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'foo' created 2024-05-06T12:31:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3748171226' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T12:31:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:37.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:37.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:37 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:37.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3748171226' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T12:31:37.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3748171226' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:37.375 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:37.376 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:37.376 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:37 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:37.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'rbd pool init foo' 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[27137]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 677 B/s wr, 1 op/s 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[27137]: Saving service rgw.foo spec with placement count:2 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[27137]: Deploying daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3748171226' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[27137]: osdmap e71: 8 total, 8 up, 8 in 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[27137]: osdmap e72: 8 total, 8 up, 8 in 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[22413]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 677 B/s wr, 1 op/s 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[22413]: Saving service rgw.foo spec with placement count:2 2024-05-06T12:31:38.338 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[22413]: Deploying daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:31:38.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3748171226' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-06T12:31:38.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[22413]: osdmap e71: 8 total, 8 up, 8 in 2024-05-06T12:31:38.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:38 smithi067 ceph-mon[22413]: osdmap e72: 8 total, 8 up, 8 in 2024-05-06T12:31:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:38 smithi086 ceph-mon[23164]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 677 B/s wr, 1 op/s 2024-05-06T12:31:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:38 smithi086 ceph-mon[23164]: Saving service rgw.foo spec with placement count:2 2024-05-06T12:31:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:38 smithi086 ceph-mon[23164]: Deploying daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:31:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:38 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3748171226' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-06T12:31:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:38 smithi086 ceph-mon[23164]: osdmap e71: 8 total, 8 up, 8 in 2024-05-06T12:31:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:38 smithi086 ceph-mon[23164]: osdmap e72: 8 total, 8 up, 8 in 2024-05-06T12:31:40.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:40 smithi067 ceph-mon[22413]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:40.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:40 smithi067 ceph-mon[22413]: osdmap e73: 8 total, 8 up, 8 in 2024-05-06T12:31:40.325 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:40 smithi067 ceph-mon[27137]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:40.325 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:40 smithi067 ceph-mon[27137]: osdmap e73: 8 total, 8 up, 8 in 2024-05-06T12:31:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:40 smithi086 ceph-mon[23164]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:40.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:40 smithi086 ceph-mon[23164]: osdmap e73: 8 total, 8 up, 8 in 2024-05-06T12:31:40.922 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:31:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:40] "GET /metrics HTTP/1.1" 200 196797 "" "Prometheus/2.33.4" 2024-05-06T12:31:41.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1191776434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T12:31:41.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:41 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T12:31:41.173 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:41 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1191776434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T12:31:41.173 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:41 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T12:31:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:41 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1191776434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T12:31:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:41 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T12:31:41.788 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:31:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: osdmap e74: 8 total, 8 up, 8 in 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[22413]: Deploying daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: osdmap e74: 8 total, 8 up, 8 in 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:42 smithi067 ceph-mon[27137]: Deploying daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:31:42.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:31:42.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-06T12:31:42.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: osdmap e74: 8 total, 8 up, 8 in 2024-05-06T12:31:42.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:42.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:42.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:42.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:42.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:42.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:42 smithi086 ceph-mon[23164]: Deploying daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:31:43.460 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:43 smithi067 ceph-mon[22413]: osdmap e75: 8 total, 8 up, 8 in 2024-05-06T12:31:43.461 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:43 smithi067 ceph-mon[27137]: osdmap e75: 8 total, 8 up, 8 in 2024-05-06T12:31:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:43 smithi086 ceph-mon[23164]: osdmap e75: 8 total, 8 up, 8 in 2024-05-06T12:31:43.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch apply iscsi foo u p' 2024-05-06T12:31:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:44 smithi086 ceph-mon[23164]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 6.9 KiB/s wr, 226 op/s 2024-05-06T12:31:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:44 smithi086 ceph-mon[23164]: osdmap e76: 8 total, 8 up, 8 in 2024-05-06T12:31:44.380 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:44 smithi067 ceph-mon[22413]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 6.9 KiB/s wr, 226 op/s 2024-05-06T12:31:44.380 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:44 smithi067 ceph-mon[22413]: osdmap e76: 8 total, 8 up, 8 in 2024-05-06T12:31:44.380 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:44 smithi067 ceph-mon[27137]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 6.9 KiB/s wr, 226 op/s 2024-05-06T12:31:44.380 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:44 smithi067 ceph-mon[27137]: osdmap e76: 8 total, 8 up, 8 in 2024-05-06T12:31:46.077 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled iscsi.foo update... 2024-05-06T12:31:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[22413]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 5.7 KiB/s wr, 188 op/s 2024-05-06T12:31:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[27137]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 5.7 KiB/s wr, 188 op/s 2024-05-06T12:31:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:45 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:45 smithi086 ceph-mon[23164]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 5.7 KiB/s wr, 188 op/s 2024-05-06T12:31:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:45 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:45 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:45 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:46.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:45 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:46.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:45 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:46.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:45 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:46.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 120' 2024-05-06T12:31:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: Saving service rgw.smpl spec with placement count:2 2024-05-06T12:31:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: Deploying daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: Saving service iscsi.foo spec with placement count:1 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:31:47.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: Saving service rgw.smpl spec with placement count:2 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: Deploying daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: Saving service iscsi.foo spec with placement count:1 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:31:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: Saving service rgw.smpl spec with placement count:2 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: Deploying daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='client.24632 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: Saving service iscsi.foo spec with placement count:1 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:31:47.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:31:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[22413]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.7 KiB/s wr, 187 op/s 2024-05-06T12:31:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-05-06T12:31:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-05-06T12:31:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-05-06T12:31:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-05-06T12:31:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[22413]: osdmap e77: 8 total, 8 up, 8 in 2024-05-06T12:31:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[27137]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.7 KiB/s wr, 187 op/s 2024-05-06T12:31:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-05-06T12:31:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-05-06T12:31:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-05-06T12:31:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-05-06T12:31:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:48 smithi067 ceph-mon[27137]: osdmap e77: 8 total, 8 up, 8 in 2024-05-06T12:31:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:48 smithi086 ceph-mon[23164]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.7 KiB/s wr, 187 op/s 2024-05-06T12:31:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:48 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:48 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-05-06T12:31:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:48 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-05-06T12:31:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:48 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-05-06T12:31:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:48 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-05-06T12:31:48.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:48 smithi086 ceph-mon[23164]: osdmap e77: 8 total, 8 up, 8 in 2024-05-06T12:31:49.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:49 smithi067 ceph-mon[22413]: osdmap e78: 8 total, 8 up, 8 in 2024-05-06T12:31:49.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:49 smithi067 ceph-mon[27137]: osdmap e78: 8 total, 8 up, 8 in 2024-05-06T12:31:49.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:49 smithi086 ceph-mon[23164]: osdmap e78: 8 total, 8 up, 8 in 2024-05-06T12:31:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:50 smithi086 ceph-mon[23164]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 166 op/s 2024-05-06T12:31:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:50 smithi086 ceph-mon[23164]: osdmap e79: 8 total, 8 up, 8 in 2024-05-06T12:31:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:50 smithi067 ceph-mon[22413]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 166 op/s 2024-05-06T12:31:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:50 smithi067 ceph-mon[22413]: osdmap e79: 8 total, 8 up, 8 in 2024-05-06T12:31:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:50 smithi067 ceph-mon[27137]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 166 op/s 2024-05-06T12:31:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:50 smithi067 ceph-mon[27137]: osdmap e79: 8 total, 8 up, 8 in 2024-05-06T12:31:50.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:31:50 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:50] "GET /metrics HTTP/1.1" 200 207150 "" "Prometheus/2.33.4" 2024-05-06T12:31:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:51.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:51.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:51.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:51 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:51.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:51 smithi086 ceph-mon[23164]: Deploying daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:31:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:51.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[22413]: Deploying daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:31:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:31:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T12:31:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:51.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:51 smithi067 ceph-mon[27137]: Deploying daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:31:51.771 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:31:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:31:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:31:52.478 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:52 smithi086 ceph-mon[23164]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 166 op/s 2024-05-06T12:31:52.478 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:52 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:52.584 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:52 smithi067 ceph-mon[22413]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 166 op/s 2024-05-06T12:31:52.584 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:52 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:52.584 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:52 smithi067 ceph-mon[27137]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 166 op/s 2024-05-06T12:31:52.585 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:52 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:53.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:53 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:53.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:53 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:53.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:53 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:53.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:53 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:53.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:53 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:53.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:53 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:53.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:53 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:53.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:53 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:53.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:53 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:54.815 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:54 smithi067 ceph-mon[22413]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 324 KiB/s rd, 682 B/s wr, 501 op/s; 7 B/s, 0 objects/s recovering 2024-05-06T12:31:54.816 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:54 smithi067 ceph-mon[27137]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 324 KiB/s rd, 682 B/s wr, 501 op/s; 7 B/s, 0 objects/s recovering 2024-05-06T12:31:54.831 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:54 smithi086 ceph-mon[23164]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 324 KiB/s rd, 682 B/s wr, 501 op/s; 7 B/s, 0 objects/s recovering 2024-05-06T12:31:56.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 516 B/s wr, 379 op/s; 5 B/s, 0 objects/s recovering 2024-05-06T12:31:56.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:31:56.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-06T12:31:56.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:56 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:56.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 516 B/s wr, 379 op/s; 5 B/s, 0 objects/s recovering 2024-05-06T12:31:56.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 516 B/s wr, 379 op/s; 5 B/s, 0 objects/s recovering 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-06T12:31:56.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:56 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:57.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:57 smithi086 ceph-mon[23164]: Checking dashboard <-> RGW credentials 2024-05-06T12:31:57.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:57 smithi086 ceph-mon[23164]: Checking pool "foo" exists for service iscsi.foo 2024-05-06T12:31:57.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:57 smithi086 ceph-mon[23164]: Deploying daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:31:57.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:57 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:57.655 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[22413]: Checking dashboard <-> RGW credentials 2024-05-06T12:31:57.655 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[22413]: Checking pool "foo" exists for service iscsi.foo 2024-05-06T12:31:57.655 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[22413]: Deploying daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:31:57.656 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:57.656 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[27137]: Checking dashboard <-> RGW credentials 2024-05-06T12:31:57.656 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[27137]: Checking pool "foo" exists for service iscsi.foo 2024-05-06T12:31:57.656 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[27137]: Deploying daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:31:57.656 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:57 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:58.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:58 smithi086 ceph-mon[23164]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 229 B/s wr, 224 op/s; 5 B/s, 0 objects/s recovering 2024-05-06T12:31:58.694 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:58 smithi067 ceph-mon[22413]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 229 B/s wr, 224 op/s; 5 B/s, 0 objects/s recovering 2024-05-06T12:31:58.694 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:58 smithi067 ceph-mon[27137]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 229 B/s wr, 224 op/s; 5 B/s, 0 objects/s recovering 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[22413]: mgrmap e21: y(active, since 2m), standbys: x 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[27137]: mgrmap e21: y(active, since 2m), standbys: x 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:59.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:31:59 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:31:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:59 smithi086 ceph-mon[23164]: mgrmap e21: y(active, since 2m), standbys: x 2024-05-06T12:31:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:31:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:31:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:31:59 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:32:00.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:32:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:00] "GET /metrics HTTP/1.1" 200 207150 "" "Prometheus/2.33.4" 2024-05-06T12:32:00.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:00 smithi067 ceph-mon[22413]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 409 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-05-06T12:32:00.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:00 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1054159638' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:32:00.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:00 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/347311268' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2422689785"}]: dispatch 2024-05-06T12:32:00.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:00 smithi067 ceph-mon[27137]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 409 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-05-06T12:32:00.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:00 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1054159638' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:32:00.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:00 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/347311268' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2422689785"}]: dispatch 2024-05-06T12:32:00.794 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:00 smithi086 ceph-mon[23164]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 409 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-05-06T12:32:00.794 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:00 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1054159638' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:32:00.795 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:00 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/347311268' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2422689785"}]: dispatch 2024-05-06T12:32:01.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:32:01 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/347311268' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2422689785"}]': finished 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[27137]: osdmap e80: 8 total, 8 up, 8 in 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[27137]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 409 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1437019202' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]: dispatch 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]: dispatch 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/347311268' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2422689785"}]': finished 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[22413]: osdmap e80: 8 total, 8 up, 8 in 2024-05-06T12:32:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[22413]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 409 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-05-06T12:32:02.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1437019202' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]: dispatch 2024-05-06T12:32:02.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:01 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]: dispatch 2024-05-06T12:32:02.334 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:01 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/347311268' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2422689785"}]': finished 2024-05-06T12:32:02.334 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:01 smithi086 ceph-mon[23164]: osdmap e80: 8 total, 8 up, 8 in 2024-05-06T12:32:02.335 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:01 smithi086 ceph-mon[23164]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 409 B/s wr, 205 op/s; 4 B/s, 0 objects/s recovering 2024-05-06T12:32:02.335 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:01 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1437019202' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]: dispatch 2024-05-06T12:32:02.335 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:01 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]: dispatch 2024-05-06T12:32:03.118 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]': finished 2024-05-06T12:32:03.118 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: osdmap e81: 8 total, 8 up, 8 in 2024-05-06T12:32:03.118 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.118 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1247497599' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]: dispatch 2024-05-06T12:32:03.118 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]: dispatch 2024-05-06T12:32:03.118 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.118 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi067"}]: dispatch 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]': finished 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: osdmap e81: 8 total, 8 up, 8 in 2024-05-06T12:32:03.119 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1247497599' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]: dispatch 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]: dispatch 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi067"}]: dispatch 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:32:03.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:02 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:32:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1588658531"}]': finished 2024-05-06T12:32:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: osdmap e81: 8 total, 8 up, 8 in 2024-05-06T12:32:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1247497599' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]: dispatch 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]: dispatch 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi067"}]: dispatch 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:32:03.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:02 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:32:03.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:32:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:32:03.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:32:03.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:32:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:32:03.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:32:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T12:32:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: Adding iSCSI gateway http://:@172.21.15.67:5000 to Dashboard 2024-05-06T12:32:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T12:32:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi067"}]: dispatch 2024-05-06T12:32:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-05-06T12:32:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]': finished 2024-05-06T12:32:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: osdmap e82: 8 total, 8 up, 8 in 2024-05-06T12:32:04.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:04 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1823055625' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/4180882082"}]: dispatch 2024-05-06T12:32:04.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T12:32:04.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: Adding iSCSI gateway http://:@172.21.15.67:5000 to Dashboard 2024-05-06T12:32:04.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T12:32:04.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi067"}]: dispatch 2024-05-06T12:32:04.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-05-06T12:32:04.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]': finished 2024-05-06T12:32:04.424 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: osdmap e82: 8 total, 8 up, 8 in 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1823055625' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/4180882082"}]: dispatch 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: Adding iSCSI gateway http://:@172.21.15.67:5000 to Dashboard 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi067"}]: dispatch 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 383 B/s wr, 7 op/s 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/4062172122"}]': finished 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: osdmap e82: 8 total, 8 up, 8 in 2024-05-06T12:32:04.425 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:04 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1823055625' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/4180882082"}]: dispatch 2024-05-06T12:32:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:05 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1823055625' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/4180882082"}]': finished 2024-05-06T12:32:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:05 smithi086 ceph-mon[23164]: osdmap e83: 8 total, 8 up, 8 in 2024-05-06T12:32:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:05 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/229844889' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]: dispatch 2024-05-06T12:32:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:05 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]: dispatch 2024-05-06T12:32:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1823055625' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/4180882082"}]': finished 2024-05-06T12:32:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[22413]: osdmap e83: 8 total, 8 up, 8 in 2024-05-06T12:32:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/229844889' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]: dispatch 2024-05-06T12:32:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]: dispatch 2024-05-06T12:32:05.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1823055625' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/4180882082"}]': finished 2024-05-06T12:32:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[27137]: osdmap e83: 8 total, 8 up, 8 in 2024-05-06T12:32:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/229844889' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]: dispatch 2024-05-06T12:32:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:05 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]: dispatch 2024-05-06T12:32:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:06 smithi086 ceph-mon[23164]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1016 B/s rd, 254 B/s wr, 3 op/s 2024-05-06T12:32:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:06 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]': finished 2024-05-06T12:32:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:06 smithi086 ceph-mon[23164]: osdmap e84: 8 total, 8 up, 8 in 2024-05-06T12:32:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:06 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/952639978' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]: dispatch 2024-05-06T12:32:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:06 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]: dispatch 2024-05-06T12:32:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:06 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[27137]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1016 B/s rd, 254 B/s wr, 3 op/s 2024-05-06T12:32:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]': finished 2024-05-06T12:32:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[27137]: osdmap e84: 8 total, 8 up, 8 in 2024-05-06T12:32:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/952639978' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]: dispatch 2024-05-06T12:32:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]: dispatch 2024-05-06T12:32:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:06.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[22413]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1016 B/s rd, 254 B/s wr, 3 op/s 2024-05-06T12:32:06.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2195830361"}]': finished 2024-05-06T12:32:06.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[22413]: osdmap e84: 8 total, 8 up, 8 in 2024-05-06T12:32:06.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/952639978' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]: dispatch 2024-05-06T12:32:06.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]: dispatch 2024-05-06T12:32:06.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:06 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:07.392 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]': finished 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[27137]: osdmap e85: 8 total, 8 up, 8 in 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[27137]: Checking dashboard <-> RGW credentials 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2172960654' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]: dispatch 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]: dispatch 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]': finished 2024-05-06T12:32:07.393 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[22413]: osdmap e85: 8 total, 8 up, 8 in 2024-05-06T12:32:07.394 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:07.394 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[22413]: Checking dashboard <-> RGW credentials 2024-05-06T12:32:07.394 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2172960654' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]: dispatch 2024-05-06T12:32:07.394 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]: dispatch 2024-05-06T12:32:07.394 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:07 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:07.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:07 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/4180882082"}]': finished 2024-05-06T12:32:07.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:07 smithi086 ceph-mon[23164]: osdmap e85: 8 total, 8 up, 8 in 2024-05-06T12:32:07.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:07 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:07.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:07 smithi086 ceph-mon[23164]: Checking dashboard <-> RGW credentials 2024-05-06T12:32:07.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:07 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2172960654' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]: dispatch 2024-05-06T12:32:07.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:07 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]: dispatch 2024-05-06T12:32:07.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:07 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[22413]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]': finished 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[22413]: osdmap e86: 8 total, 8 up, 8 in 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/4180077524' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/712625767"}]: dispatch 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[27137]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]': finished 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[27137]: osdmap e86: 8 total, 8 up, 8 in 2024-05-06T12:32:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:08 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/4180077524' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/712625767"}]: dispatch 2024-05-06T12:32:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:08 smithi086 ceph-mon[23164]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:32:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:08 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/388484257"}]': finished 2024-05-06T12:32:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:08 smithi086 ceph-mon[23164]: osdmap e86: 8 total, 8 up, 8 in 2024-05-06T12:32:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:08 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/4180077524' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/712625767"}]: dispatch 2024-05-06T12:32:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:09 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/4180077524' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/712625767"}]': finished 2024-05-06T12:32:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:09 smithi086 ceph-mon[23164]: osdmap e87: 8 total, 8 up, 8 in 2024-05-06T12:32:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:09 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2803517394' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]: dispatch 2024-05-06T12:32:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:09 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]: dispatch 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/4180077524' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/712625767"}]': finished 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[27137]: osdmap e87: 8 total, 8 up, 8 in 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2803517394' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]: dispatch 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]: dispatch 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/4180077524' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/712625767"}]': finished 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[22413]: osdmap e87: 8 total, 8 up, 8 in 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2803517394' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]: dispatch 2024-05-06T12:32:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:09 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]: dispatch 2024-05-06T12:32:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:10 smithi086 ceph-mon[23164]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T12:32:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:10 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:10 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]': finished 2024-05-06T12:32:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:10 smithi086 ceph-mon[23164]: osdmap e88: 8 total, 8 up, 8 in 2024-05-06T12:32:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:10 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2833814817' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]: dispatch 2024-05-06T12:32:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:10 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]: dispatch 2024-05-06T12:32:10.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:32:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:10] "GET /metrics HTTP/1.1" 200 214920 "" "Prometheus/2.33.4" 2024-05-06T12:32:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[22413]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T12:32:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]': finished 2024-05-06T12:32:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[22413]: osdmap e88: 8 total, 8 up, 8 in 2024-05-06T12:32:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2833814817' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]: dispatch 2024-05-06T12:32:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]: dispatch 2024-05-06T12:32:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[27137]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T12:32:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2204009303"}]': finished 2024-05-06T12:32:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[27137]: osdmap e88: 8 total, 8 up, 8 in 2024-05-06T12:32:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2833814817' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]: dispatch 2024-05-06T12:32:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:10 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]: dispatch 2024-05-06T12:32:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:11 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]': finished 2024-05-06T12:32:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:11 smithi086 ceph-mon[23164]: osdmap e89: 8 total, 8 up, 8 in 2024-05-06T12:32:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:11 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/41575225' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1200758237"}]: dispatch 2024-05-06T12:32:11.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:32:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:32:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:11 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]': finished 2024-05-06T12:32:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:11 smithi067 ceph-mon[22413]: osdmap e89: 8 total, 8 up, 8 in 2024-05-06T12:32:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:11 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/41575225' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1200758237"}]: dispatch 2024-05-06T12:32:11.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:11 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586832651"}]': finished 2024-05-06T12:32:11.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:11 smithi067 ceph-mon[27137]: osdmap e89: 8 total, 8 up, 8 in 2024-05-06T12:32:11.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:11 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/41575225' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1200758237"}]: dispatch 2024-05-06T12:32:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:12 smithi086 ceph-mon[23164]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T12:32:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:12 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/41575225' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1200758237"}]': finished 2024-05-06T12:32:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:12 smithi086 ceph-mon[23164]: osdmap e90: 8 total, 8 up, 8 in 2024-05-06T12:32:12.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:12 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2945751976' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1588658531"}]: dispatch 2024-05-06T12:32:12.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[22413]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T12:32:12.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/41575225' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1200758237"}]': finished 2024-05-06T12:32:12.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[22413]: osdmap e90: 8 total, 8 up, 8 in 2024-05-06T12:32:12.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2945751976' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1588658531"}]: dispatch 2024-05-06T12:32:12.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[27137]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T12:32:12.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/41575225' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1200758237"}]': finished 2024-05-06T12:32:12.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[27137]: osdmap e90: 8 total, 8 up, 8 in 2024-05-06T12:32:12.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:12 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2945751976' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1588658531"}]: dispatch 2024-05-06T12:32:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:13 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2945751976' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1588658531"}]': finished 2024-05-06T12:32:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:13 smithi086 ceph-mon[23164]: osdmap e91: 8 total, 8 up, 8 in 2024-05-06T12:32:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:13 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3801390756' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1200758237"}]: dispatch 2024-05-06T12:32:13.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:13 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2945751976' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1588658531"}]': finished 2024-05-06T12:32:13.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:13 smithi067 ceph-mon[27137]: osdmap e91: 8 total, 8 up, 8 in 2024-05-06T12:32:13.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:13 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3801390756' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1200758237"}]: dispatch 2024-05-06T12:32:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:13 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2945751976' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1588658531"}]': finished 2024-05-06T12:32:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:13 smithi067 ceph-mon[22413]: osdmap e91: 8 total, 8 up, 8 in 2024-05-06T12:32:13.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:13 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3801390756' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1200758237"}]: dispatch 2024-05-06T12:32:13.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:32:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:32:13.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:32:13.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:32:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:32:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:32:13.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:32:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:32:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:32:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:14 smithi086 ceph-mon[23164]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:14 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3801390756' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1200758237"}]': finished 2024-05-06T12:32:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:14 smithi086 ceph-mon[23164]: osdmap e92: 8 total, 8 up, 8 in 2024-05-06T12:32:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:14 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2808539677' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1872361456"}]: dispatch 2024-05-06T12:32:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[22413]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3801390756' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1200758237"}]': finished 2024-05-06T12:32:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[22413]: osdmap e92: 8 total, 8 up, 8 in 2024-05-06T12:32:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2808539677' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1872361456"}]: dispatch 2024-05-06T12:32:14.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[27137]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:14.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3801390756' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1200758237"}]': finished 2024-05-06T12:32:14.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[27137]: osdmap e92: 8 total, 8 up, 8 in 2024-05-06T12:32:14.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:14 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2808539677' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1872361456"}]: dispatch 2024-05-06T12:32:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:15 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2808539677' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1872361456"}]': finished 2024-05-06T12:32:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:15 smithi086 ceph-mon[23164]: osdmap e93: 8 total, 8 up, 8 in 2024-05-06T12:32:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:15 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2580512772' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3823878302"}]: dispatch 2024-05-06T12:32:15.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:15 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2808539677' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1872361456"}]': finished 2024-05-06T12:32:15.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:15 smithi067 ceph-mon[27137]: osdmap e93: 8 total, 8 up, 8 in 2024-05-06T12:32:15.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:15 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2580512772' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3823878302"}]: dispatch 2024-05-06T12:32:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:15 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2808539677' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1872361456"}]': finished 2024-05-06T12:32:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:15 smithi067 ceph-mon[22413]: osdmap e93: 8 total, 8 up, 8 in 2024-05-06T12:32:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:15 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2580512772' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3823878302"}]: dispatch 2024-05-06T12:32:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:16 smithi086 ceph-mon[23164]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:16 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2580512772' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3823878302"}]': finished 2024-05-06T12:32:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:16 smithi086 ceph-mon[23164]: osdmap e94: 8 total, 8 up, 8 in 2024-05-06T12:32:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:16 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1177666907' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]: dispatch 2024-05-06T12:32:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:16 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]: dispatch 2024-05-06T12:32:16.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[22413]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:16.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2580512772' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3823878302"}]': finished 2024-05-06T12:32:16.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[22413]: osdmap e94: 8 total, 8 up, 8 in 2024-05-06T12:32:16.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1177666907' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]: dispatch 2024-05-06T12:32:16.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]: dispatch 2024-05-06T12:32:16.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[27137]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:16.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2580512772' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3823878302"}]': finished 2024-05-06T12:32:16.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[27137]: osdmap e94: 8 total, 8 up, 8 in 2024-05-06T12:32:16.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1177666907' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]: dispatch 2024-05-06T12:32:16.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:16 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]: dispatch 2024-05-06T12:32:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:17 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]': finished 2024-05-06T12:32:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:17 smithi086 ceph-mon[23164]: osdmap e95: 8 total, 8 up, 8 in 2024-05-06T12:32:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:17 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]': finished 2024-05-06T12:32:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:17 smithi067 ceph-mon[22413]: osdmap e95: 8 total, 8 up, 8 in 2024-05-06T12:32:17.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:17 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1500831993"}]': finished 2024-05-06T12:32:17.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:17 smithi067 ceph-mon[27137]: osdmap e95: 8 total, 8 up, 8 in 2024-05-06T12:32:18.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:18 smithi086 ceph-mon[23164]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:32:18.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:18 smithi067 ceph-mon[22413]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:32:18.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:18 smithi067 ceph-mon[27137]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:32:20.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:20 smithi067 ceph-mon[22413]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-05-06T12:32:20.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:20 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:20.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:32:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:20] "GET /metrics HTTP/1.1" 200 214935 "" "Prometheus/2.33.4" 2024-05-06T12:32:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:20 smithi067 ceph-mon[27137]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-05-06T12:32:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:20 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:21.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:20 smithi086 ceph-mon[23164]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-05-06T12:32:21.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:20 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:21.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:32:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:32:22.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:22 smithi067 ceph-mon[22413]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-05-06T12:32:22.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:22 smithi067 ceph-mon[27137]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-05-06T12:32:23.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:22 smithi086 ceph-mon[23164]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-05-06T12:32:23.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:32:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:32:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:32:24.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:24 smithi067 ceph-mon[22413]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:24.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:24 smithi067 ceph-mon[27137]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:25.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:24 smithi086 ceph-mon[23164]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:26.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:25 smithi086 ceph-mon[23164]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:32:26.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:25 smithi067 ceph-mon[27137]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:32:26.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:25 smithi067 ceph-mon[22413]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:32:28.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:28 smithi086 ceph-mon[23164]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-05-06T12:32:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:28 smithi067 ceph-mon[27137]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-05-06T12:32:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:28 smithi067 ceph-mon[22413]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-05-06T12:32:30.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:30 smithi086 ceph-mon[23164]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:30.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:30 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:30 smithi067 ceph-mon[22413]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:30 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:30 smithi067 ceph-mon[27137]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:30 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:30.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:32:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:30] "GET /metrics HTTP/1.1" 200 214935 "" "Prometheus/2.33.4" 2024-05-06T12:32:31.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:32:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:32:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:32 smithi086 ceph-mon[23164]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:32.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:32 smithi067 ceph-mon[22413]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:32.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:32 smithi067 ceph-mon[27137]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:34 smithi086 ceph-mon[23164]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:34 smithi067 ceph-mon[22413]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:34.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:34 smithi067 ceph-mon[27137]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:36.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:36 smithi086 ceph-mon[23164]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:36.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:36 smithi067 ceph-mon[22413]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:36.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:36 smithi067 ceph-mon[27137]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:38 smithi086 ceph-mon[23164]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:38.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:38 smithi067 ceph-mon[22413]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:38.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:38 smithi067 ceph-mon[27137]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:40 smithi086 ceph-mon[23164]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:40 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:40 smithi067 ceph-mon[27137]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:40 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:40.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:40 smithi067 ceph-mon[22413]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:40.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:40 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:40.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:32:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:40] "GET /metrics HTTP/1.1" 200 214883 "" "Prometheus/2.33.4" 2024-05-06T12:32:41.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:32:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:32:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:42 smithi086 ceph-mon[23164]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:42 smithi067 ceph-mon[22413]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:42.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:42 smithi067 ceph-mon[27137]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:44 smithi086 ceph-mon[23164]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:44 smithi067 ceph-mon[22413]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:44.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:44 smithi067 ceph-mon[27137]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:46 smithi086 ceph-mon[23164]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:46.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:46 smithi067 ceph-mon[22413]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:46.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:46 smithi067 ceph-mon[27137]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:32:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:32:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:32:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:32:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:32:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:32:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:48 smithi086 ceph-mon[23164]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:48 smithi067 ceph-mon[27137]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:48 smithi067 ceph-mon[22413]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:50 smithi086 ceph-mon[23164]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:50 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:50 smithi067 ceph-mon[22413]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:50 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:50 smithi067 ceph-mon[27137]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:50 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:32:50.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:32:50 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:50] "GET /metrics HTTP/1.1" 200 214844 "" "Prometheus/2.33.4" 2024-05-06T12:32:51.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:32:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:32:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:32:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:52 smithi086 ceph-mon[23164]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:52 smithi067 ceph-mon[22413]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:52.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:52 smithi067 ceph-mon[27137]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:54.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:54 smithi086 ceph-mon[23164]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:54.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:54 smithi067 ceph-mon[22413]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:54.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:54 smithi067 ceph-mon[27137]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:32:56.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:56 smithi086 ceph-mon[23164]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:56.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:56 smithi067 ceph-mon[22413]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:56.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:56 smithi067 ceph-mon[27137]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:58.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:32:58 smithi086 ceph-mon[23164]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:58.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:32:58 smithi067 ceph-mon[22413]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:32:58.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:32:58 smithi067 ceph-mon[27137]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:00 smithi086 ceph-mon[23164]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:00 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:00.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:00 smithi067 ceph-mon[27137]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:00.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:00 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:00.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:00 smithi067 ceph-mon[22413]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:00.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:00 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:00.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:33:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:00] "GET /metrics HTTP/1.1" 200 214844 "" "Prometheus/2.33.4" 2024-05-06T12:33:01.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:33:01 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:33:02.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:02 smithi086 ceph-mon[23164]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:02 smithi067 ceph-mon[22413]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:02.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:02 smithi067 ceph-mon[27137]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:04 smithi086 ceph-mon[23164]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:04 smithi067 ceph-mon[22413]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:04 smithi067 ceph-mon[27137]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:06 smithi086 ceph-mon[23164]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:06.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:06 smithi067 ceph-mon[22413]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:06 smithi067 ceph-mon[27137]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:07.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:07 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:33:07.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:07 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:33:07.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:07 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:33:07.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:07 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:33:07.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:07 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:33:07.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:07 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:33:07.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:07 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:33:07.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:07 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:33:07.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:07 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:33:08.271 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:08 smithi086 ceph-mon[23164]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:08.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:08 smithi067 ceph-mon[22413]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:08.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:08 smithi067 ceph-mon[27137]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:10.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:10 smithi086 ceph-mon[23164]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:10.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:10 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:10.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:10 smithi067 ceph-mon[22413]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:10.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:10 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:10 smithi067 ceph-mon[27137]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:10 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:10.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:33:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:10] "GET /metrics HTTP/1.1" 200 214839 "" "Prometheus/2.33.4" 2024-05-06T12:33:11.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:33:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:33:12.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:12 smithi086 ceph-mon[23164]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:12.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:12 smithi067 ceph-mon[22413]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:12.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:12 smithi067 ceph-mon[27137]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:14.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:14 smithi086 ceph-mon[23164]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:14 smithi067 ceph-mon[22413]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:14.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:14 smithi067 ceph-mon[27137]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:16.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:16 smithi086 ceph-mon[23164]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:16.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:16 smithi067 ceph-mon[22413]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:16 smithi067 ceph-mon[27137]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:18.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:18 smithi086 ceph-mon[23164]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:18.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:18 smithi067 ceph-mon[27137]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:18.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:18 smithi067 ceph-mon[22413]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:20.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:20 smithi086 ceph-mon[23164]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:20.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:20 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:20.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:20 smithi067 ceph-mon[22413]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:20.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:20 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:20.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:20 smithi067 ceph-mon[27137]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:20.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:20 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:20.974 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:33:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:20] "GET /metrics HTTP/1.1" 200 214822 "" "Prometheus/2.33.4" 2024-05-06T12:33:21.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:33:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:33:22.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:22 smithi086 ceph-mon[23164]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:22.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:22 smithi067 ceph-mon[22413]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:22.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:22 smithi067 ceph-mon[27137]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:24 smithi067 ceph-mon[22413]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:24 smithi067 ceph-mon[27137]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:24.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:24 smithi086 ceph-mon[23164]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:26.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:26 smithi086 ceph-mon[23164]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:26.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:26 smithi067 ceph-mon[27137]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:26.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:26 smithi067 ceph-mon[22413]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:28.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:28 smithi086 ceph-mon[23164]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:28 smithi067 ceph-mon[27137]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:28 smithi067 ceph-mon[22413]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:30.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:30 smithi086 ceph-mon[23164]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:30.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:30 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:30 smithi067 ceph-mon[27137]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:30 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:30 smithi067 ceph-mon[22413]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:30 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:30.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:33:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:30] "GET /metrics HTTP/1.1" 200 214822 "" "Prometheus/2.33.4" 2024-05-06T12:33:31.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:33:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:33:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:32 smithi086 ceph-mon[23164]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:32.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:32 smithi067 ceph-mon[22413]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:32.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:32 smithi067 ceph-mon[27137]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:34 smithi086 ceph-mon[23164]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:34 smithi067 ceph-mon[22413]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:34.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:34 smithi067 ceph-mon[27137]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:36.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:36 smithi086 ceph-mon[23164]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:36.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:36 smithi067 ceph-mon[22413]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:36.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:36 smithi067 ceph-mon[27137]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:38 smithi086 ceph-mon[23164]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:38 smithi067 ceph-mon[22413]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:38.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:38 smithi067 ceph-mon[27137]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:40 smithi086 ceph-mon[23164]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:40 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:40 smithi067 ceph-mon[27137]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:40 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:40.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:40 smithi067 ceph-mon[22413]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:40.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:40 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:40.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:33:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:40] "GET /metrics HTTP/1.1" 200 214837 "" "Prometheus/2.33.4" 2024-05-06T12:33:41.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:33:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:33:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:42 smithi086 ceph-mon[23164]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:42 smithi067 ceph-mon[22413]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:42.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:42 smithi067 ceph-mon[27137]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:44 smithi086 ceph-mon[23164]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:44 smithi067 ceph-mon[22413]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:44.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:44 smithi067 ceph-mon[27137]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:46 smithi086 ceph-mon[23164]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:46.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:46 smithi067 ceph-mon[22413]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:46.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:46 smithi067 ceph-mon[27137]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:33:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:33:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:33:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:33:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:33:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:33:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:48 smithi086 ceph-mon[23164]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:48 smithi067 ceph-mon[22413]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:48 smithi067 ceph-mon[27137]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:50.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:50 smithi067 ceph-mon[22413]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:50.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:50 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:50.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:50 smithi067 ceph-mon[27137]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:50.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:50 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:50 smithi086 ceph-mon[23164]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:50 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:33:50.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-05-06T12:33:50.694 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:33:50 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:50] "GET /metrics HTTP/1.1" 200 214845 "" "Prometheus/2.33.4" 2024-05-06T12:33:51.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:33:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:33:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:33:52.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:52 smithi067 ceph-mon[22413]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:52.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:52 smithi067 ceph-mon[27137]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:52.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:52 smithi086 ceph-mon[23164]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:52.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-05-06T12:33:54.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:54 smithi086 ceph-mon[23164]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:54.391 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:54 smithi067 ceph-mon[22413]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:54.391 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:54 smithi067 ceph-mon[27137]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:33:55.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set global log_to_journald false --force' 2024-05-06T12:33:56.153 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:56 smithi067 ceph-mon[27137]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:56.153 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:56 smithi067 ceph-mon[22413]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:56.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:56 smithi086 ceph-mon[23164]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:58.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-05-06T12:33:58.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:33:58 smithi086 ceph-mon[23164]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:58.401 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:33:58 smithi067 ceph-mon[22413]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:33:58.402 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:33:58 smithi067 ceph-mon[27137]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:00.054 INFO:teuthology.orchestra.run.smithi067.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:34:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:00 smithi086 ceph-mon[23164]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:00 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:00 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:00 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:34:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:00 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:34:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:00 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:00.421 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[22413]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:00.421 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:00.421 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:00.421 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:34:00.421 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:34:00.421 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:00.421 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[27137]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:00.422 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:00.422 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:00.422 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:34:00.422 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:34:00.422 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:00 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:00.688 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:34:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:00] "GET /metrics HTTP/1.1" 200 214845 "" "Prometheus/2.33.4" 2024-05-06T12:34:00.711 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T12:34:00.720 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-06T12:34:00.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-05-06T12:34:01.327 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:01 smithi067 ceph-mon[22413]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:01.327 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:01 smithi067 ceph-mon[22413]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:34:01.327 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:01 smithi067 ceph-mon[22413]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:34:01.327 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:01 smithi067 ceph-mon[27137]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:01.328 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:01 smithi067 ceph-mon[27137]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:34:01.328 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:01 smithi067 ceph-mon[27137]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:34:01.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:01 smithi086 ceph-mon[23164]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:01.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:01 smithi086 ceph-mon[23164]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:34:01.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:01 smithi086 ceph-mon[23164]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:34:01.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:34:01 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:34:02.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:02 smithi067 ceph-mon[22413]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:02.303 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:02 smithi067 ceph-mon[27137]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:02.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:02 smithi086 ceph-mon[23164]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:02.763 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:34:03.414 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:34:03.414 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (2m) 117s ago 3m 15.8M - ba2b418f427c f43d96904837 2024-05-06T12:34:03.414 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (2m) 117s ago 2m 47.4M - 8.3.5 dad864ee21e9 87f103efe219 2024-05-06T12:34:03.414 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (2m) 117s ago 2m 52.1M - 3.5 e1d6a67b021e 1a04cb514a79 2024-05-06T12:34:03.414 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443 running (7m) 117s ago 7m 411M - 17.2.0 e1d6a67b021e baa64463db03 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:9283 running (9m) 117s ago 9m 455M - 17.2.0 e1d6a67b021e 156e546a6ad8 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (9m) 117s ago 9m 46.3M 2048M 17.2.0 e1d6a67b021e f5b02626cc9d 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (7m) 117s ago 7m 40.3M 2048M 17.2.0 e1d6a67b021e 5cb539dec1ee 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (8m) 117s ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e d0561ab1fc3c 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (3m) 117s ago 3m 15.8M - 1dbe0e931976 50bfd17cec65 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (3m) 117s ago 3m 14.0M - 1dbe0e931976 b88ef5b0017c 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (7m) 117s ago 7m 49.1M 4016M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (7m) 117s ago 7m 49.6M 4016M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (6m) 117s ago 6m 44.7M 4016M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (6m) 117s ago 6m 47.9M 4016M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (5m) 117s ago 5m 48.2M 4272M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:34:03.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (5m) 117s ago 5m 44.8M 4272M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:34:03.416 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (5m) 117s ago 5m 47.3M 4272M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:34:03.416 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (4m) 117s ago 4m 49.9M 4272M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:34:03.416 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (2m) 117s ago 3m 43.7M - 514e6a882f6e af867ca274de 2024-05-06T12:34:03.416 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (2m) 117s ago 2m 71.1M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:34:03.416 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (2m) 117s ago 2m 69.3M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:34:03.416 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (2m) 117s ago 2m 71.2M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:34:03.416 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (2m) 117s ago 2m 69.3M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:34:03.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:03 smithi067 ceph-mon[22413]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:03.417 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:03 smithi067 ceph-mon[27137]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:03.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:03 smithi086 ceph-mon[23164]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:03.827 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": {}, 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:34:03.828 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-05-06T12:34:03.829 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:34:03.829 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:34:04.191 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:34:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:34:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:34:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-05-06T12:34:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "", 2024-05-06T12:34:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa image" 2024-05-06T12:34:04.191 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[22413]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[22413]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[22413]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1557944824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[27137]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[27137]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[27137]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:04 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1557944824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:34:04.610 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:34:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:04 smithi086 ceph-mon[23164]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:04 smithi086 ceph-mon[23164]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:04 smithi086 ceph-mon[23164]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:04 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1557944824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:34:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:05 smithi067 ceph-mon[22413]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:05 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1762179234' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:34:05.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:05 smithi067 ceph-mon[27137]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:05.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:05 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1762179234' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:34:05.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:05 smithi086 ceph-mon[23164]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:05.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:05 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1762179234' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:34:06.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:06 smithi067 ceph-mon[22413]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:06 smithi067 ceph-mon[27137]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:06.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:06 smithi086 ceph-mon[23164]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:08.421 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:08 smithi086 ceph-mon[23164]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:08.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:08 smithi067 ceph-mon[27137]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:08.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:08 smithi067 ceph-mon[22413]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:10.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:10 smithi067 ceph-mon[22413]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:10.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:10 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:10 smithi067 ceph-mon[27137]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:10.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:10 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:10 smithi086 ceph-mon[23164]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:10 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:10.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:34:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:10] "GET /metrics HTTP/1.1" 200 214836 "" "Prometheus/2.33.4" 2024-05-06T12:34:11.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:34:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:34:12.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:12 smithi067 ceph-mon[22413]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:12.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:12 smithi067 ceph-mon[27137]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:12 smithi086 ceph-mon[23164]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:14 smithi067 ceph-mon[22413]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:14.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:14 smithi067 ceph-mon[27137]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:14.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:14 smithi086 ceph-mon[23164]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:16.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:16 smithi086 ceph-mon[23164]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:16.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:16 smithi067 ceph-mon[22413]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:16 smithi067 ceph-mon[27137]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:18.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:18 smithi086 ceph-mon[23164]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:18.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:18 smithi067 ceph-mon[22413]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:18.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:18 smithi067 ceph-mon[27137]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:20.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:20 smithi086 ceph-mon[23164]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:20.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:20 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:20.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:20 smithi067 ceph-mon[22413]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:20.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:20 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:20.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:20 smithi067 ceph-mon[27137]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:20.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:20 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:20.974 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:34:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:20] "GET /metrics HTTP/1.1" 200 214806 "" "Prometheus/2.33.4" 2024-05-06T12:34:21.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:34:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:34:22.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:22 smithi086 ceph-mon[23164]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:22.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:22 smithi067 ceph-mon[22413]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:22.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:22 smithi067 ceph-mon[27137]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:24.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:24 smithi086 ceph-mon[23164]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:24.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:24 smithi067 ceph-mon[22413]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:24.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:24 smithi067 ceph-mon[27137]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:26.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:26 smithi086 ceph-mon[23164]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:26.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:26 smithi067 ceph-mon[22413]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:26.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:26 smithi067 ceph-mon[27137]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:28.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:28 smithi086 ceph-mon[23164]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:28.474 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:28 smithi067 ceph-mon[27137]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:28 smithi067 ceph-mon[22413]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:30.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:30 smithi086 ceph-mon[23164]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:30.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:30 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:30 smithi067 ceph-mon[22413]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:30 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:30 smithi067 ceph-mon[27137]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:30 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:30.974 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:34:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:30] "GET /metrics HTTP/1.1" 200 214806 "" "Prometheus/2.33.4" 2024-05-06T12:34:31.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:34:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:34:32.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:32 smithi086 ceph-mon[23164]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:32.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:32 smithi067 ceph-mon[22413]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:32.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:32 smithi067 ceph-mon[27137]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:34.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:34 smithi067 ceph-mon[22413]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:34.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:34 smithi067 ceph-mon[27137]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:34.474 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:34 smithi086 ceph-mon[23164]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:34.937 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (3m) 2m ago 3m 15.8M - ba2b418f427c f43d96904837 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (3m) 2m ago 3m 47.4M - 8.3.5 dad864ee21e9 87f103efe219 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (2m) 2m ago 2m 52.1M - 3.5 e1d6a67b021e 1a04cb514a79 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e baa64463db03 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 156e546a6ad8 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (9m) 2m ago 10m 46.3M 2048M 17.2.0 e1d6a67b021e f5b02626cc9d 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (8m) 2m ago 8m 40.3M 2048M 17.2.0 e1d6a67b021e 5cb539dec1ee 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (8m) 2m ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e d0561ab1fc3c 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (4m) 2m ago 4m 15.8M - 1dbe0e931976 50bfd17cec65 2024-05-06T12:34:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (4m) 2m ago 4m 14.0M - 1dbe0e931976 b88ef5b0017c 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (7m) 2m ago 7m 49.1M 4016M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (7m) 2m ago 7m 49.6M 4016M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (7m) 2m ago 7m 44.7M 4016M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (6m) 2m ago 6m 47.9M 4016M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (6m) 2m ago 6m 48.2M 4272M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (5m) 2m ago 5m 44.8M 4272M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (5m) 2m ago 5m 47.3M 4272M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (5m) 2m ago 5m 49.9M 4272M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (3m) 2m ago 4m 43.7M - 514e6a882f6e af867ca274de 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (2m) 2m ago 2m 71.1M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (2m) 2m ago 2m 69.3M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:34:35.548 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (2m) 2m ago 2m 71.2M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:34:35.549 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (2m) 2m ago 2m 69.3M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:34:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[22413]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (unknown) 2024-05-06T12:34:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[22413]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T12:34:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:34:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[22413]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-06T12:34:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[22413]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa on smithi086 2024-05-06T12:34:35.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:35.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[27137]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (unknown) 2024-05-06T12:34:35.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[27137]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T12:34:35.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:34:35.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:35.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[27137]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-06T12:34:35.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:35 smithi067 ceph-mon[27137]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa on smithi086 2024-05-06T12:34:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:35 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:35 smithi086 ceph-mon[23164]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (unknown) 2024-05-06T12:34:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:35 smithi086 ceph-mon[23164]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T12:34:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:35 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:34:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:35 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:34:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:35 smithi086 ceph-mon[23164]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-06T12:34:35.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:35 smithi086 ceph-mon[23164]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa on smithi086 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": {}, 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:34:35.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:34:35.912 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:34:35.912 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-05-06T12:34:35.912 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:34:35.912 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:34:36.227 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:34:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:34:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:34:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-05-06T12:34:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "0/23 daemons upgraded", 2024-05-06T12:34:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa image on host smithi086" 2024-05-06T12:34:36.227 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:34:36.591 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:34:36.591 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[27137]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.591 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[27137]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:36.591 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[27137]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.591 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[27137]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.591 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1184020485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:34:36.591 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[22413]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.592 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[22413]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:36.592 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[22413]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.592 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[22413]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.592 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:36 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1184020485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:34:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:36 smithi086 ceph-mon[23164]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:36 smithi086 ceph-mon[23164]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:36 smithi086 ceph-mon[23164]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:36 smithi086 ceph-mon[23164]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:36 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1184020485' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:34:37.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:37 smithi067 ceph-mon[22413]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:37.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:37 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1425263382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:34:37.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:37 smithi067 ceph-mon[27137]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:37.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:37 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1425263382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:34:37.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:37 smithi086 ceph-mon[23164]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:34:37.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:37 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1425263382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:34:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:38 smithi067 ceph-mon[22413]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:38.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:38 smithi067 ceph-mon[27137]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:38.770 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:38 smithi086 ceph-mon[23164]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:40 smithi067 ceph-mon[22413]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:40 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:40.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:34:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:40] "GET /metrics HTTP/1.1" 200 214777 "" "Prometheus/2.33.4" 2024-05-06T12:34:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:40 smithi067 ceph-mon[27137]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:40 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:40.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:40 smithi086 ceph-mon[23164]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:40.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:40 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:41.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:34:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:34:42.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:42 smithi067 ceph-mon[22413]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:42.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:42 smithi067 ceph-mon[27137]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:42.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:42 smithi086 ceph-mon[23164]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:43.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:34:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:34:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:34:43.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:34:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:34:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:34:44.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:44 smithi067 ceph-mon[22413]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:44.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:44 smithi067 ceph-mon[27137]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:44.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:44 smithi086 ceph-mon[23164]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:46 smithi086 ceph-mon[23164]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:46.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:46 smithi067 ceph-mon[22413]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:46.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:46 smithi067 ceph-mon[27137]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:34:47.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:47 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:34:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:34:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:47 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:34:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:34:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:47 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:34:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:48 smithi086 ceph-mon[23164]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:48 smithi067 ceph-mon[22413]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:48 smithi067 ceph-mon[27137]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:50 smithi067 ceph-mon[22413]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:50.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:50 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:50 smithi067 ceph-mon[27137]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:50.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:50 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:50.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:50 smithi086 ceph-mon[23164]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:50.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:50 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:34:50.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:34:50 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:50] "GET /metrics HTTP/1.1" 200 214779 "" "Prometheus/2.33.4" 2024-05-06T12:34:51.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:34:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:34:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:34:52.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:52 smithi067 ceph-mon[22413]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:52.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:52 smithi067 ceph-mon[27137]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:52.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:52 smithi086 ceph-mon[23164]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:53.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:34:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:34:53.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:34:53.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:34:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:34:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:34:53.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:34:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:34:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:34:54.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:54 smithi067 ceph-mon[22413]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:54.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:54 smithi067 ceph-mon[27137]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:54.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:54 smithi086 ceph-mon[23164]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:34:56.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:56 smithi086 ceph-mon[23164]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:56.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:56 smithi067 ceph-mon[22413]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:56.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:56 smithi067 ceph-mon[27137]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:58.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:34:58 smithi067 ceph-mon[22413]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:58.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:34:58 smithi067 ceph-mon[27137]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:34:58.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:34:58 smithi086 ceph-mon[23164]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:00.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:00 smithi067 ceph-mon[22413]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:00.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:00 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:00.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:00 smithi067 ceph-mon[27137]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:00.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:00 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:00 smithi086 ceph-mon[23164]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:00 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:00.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:00] "GET /metrics HTTP/1.1" 200 214779 "" "Prometheus/2.33.4" 2024-05-06T12:35:01.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:01 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:35:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:02 smithi067 ceph-mon[22413]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:02.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:02 smithi067 ceph-mon[27137]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:02 smithi086 ceph-mon[23164]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:03.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:35:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:03.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:03.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:35:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:04 smithi067 ceph-mon[27137]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:04 smithi067 ceph-mon[22413]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:04 smithi086 ceph-mon[23164]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:06.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:06 smithi086 ceph-mon[23164]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:06.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:06 smithi067 ceph-mon[22413]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:06 smithi067 ceph-mon[27137]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:06.928 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (3m) 3m ago 4m 15.8M - ba2b418f427c f43d96904837 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (3m) 3m ago 3m 47.4M - 8.3.5 dad864ee21e9 87f103efe219 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (3m) 3m ago 3m 52.1M - 3.5 e1d6a67b021e 1a04cb514a79 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e baa64463db03 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:9283 running (10m) 3m ago 10m 455M - 17.2.0 e1d6a67b021e 156e546a6ad8 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (10m) 3m ago 10m 46.3M 2048M 17.2.0 e1d6a67b021e f5b02626cc9d 2024-05-06T12:35:07.536 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (9m) 3m ago 9m 40.3M 2048M 17.2.0 e1d6a67b021e 5cb539dec1ee 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (9m) 3m ago 9m 39.2M 2048M 17.2.0 e1d6a67b021e d0561ab1fc3c 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (4m) 3m ago 4m 15.8M - 1dbe0e931976 50bfd17cec65 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (4m) 3m ago 4m 14.0M - 1dbe0e931976 b88ef5b0017c 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (8m) 3m ago 8m 49.1M 4016M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (8m) 3m ago 8m 49.6M 4016M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (7m) 3m ago 7m 44.7M 4016M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (7m) 3m ago 7m 47.9M 4016M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (6m) 3m ago 6m 48.2M 4272M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (6m) 3m ago 6m 44.8M 4272M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (6m) 3m ago 6m 47.3M 4272M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (5m) 3m ago 5m 49.9M 4272M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (3m) 3m ago 5m 43.7M - 514e6a882f6e af867ca274de 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (3m) 3m ago 3m 71.1M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (3m) 3m ago 3m 69.3M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (3m) 3m ago 3m 71.2M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:35:07.537 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (3m) 3m ago 3m 69.3M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:35:07.900 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": {}, 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:35:07.901 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:35:08.210 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:35:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:35:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:35:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-05-06T12:35:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "0/23 daemons upgraded", 2024-05-06T12:35:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa image on host smithi086" 2024-05-06T12:35:08.211 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:35:08.211 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[27137]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.211 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[27137]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:08.211 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[27137]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.211 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[27137]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.211 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1697410579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:35:08.212 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[22413]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.212 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[22413]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:08.212 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[22413]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.212 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[22413]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.212 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:08 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1697410579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:35:08.576 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:35:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:08 smithi086 ceph-mon[23164]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:08 smithi086 ceph-mon[23164]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:08 smithi086 ceph-mon[23164]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:08 smithi086 ceph-mon[23164]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:08 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1697410579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:35:09.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:09 smithi067 ceph-mon[22413]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:09.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:09 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/131798041' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:35:09.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:09 smithi067 ceph-mon[27137]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:09.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:09 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/131798041' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:35:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:09 smithi086 ceph-mon[23164]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:09.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:09 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/131798041' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:35:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:10 smithi067 ceph-mon[27137]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:10.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:10 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:10.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:10 smithi067 ceph-mon[22413]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:10.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:10 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:10 smithi086 ceph-mon[23164]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:10 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:10.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:10 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:10] "GET /metrics HTTP/1.1" 200 214770 "" "Prometheus/2.33.4" 2024-05-06T12:35:11.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[24349]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:35:12.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:12 smithi067 ceph-mon[27137]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:12.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:12 smithi067 ceph-mon[22413]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:12 smithi086 ceph-mon[23164]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:13.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:35:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:13.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:13.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:35:14.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:14 smithi067 ceph-mon[27137]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:14 smithi067 ceph-mon[22413]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:14.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:14 smithi086 ceph-mon[23164]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:16 smithi086 ceph-mon[23164]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:16 smithi086 ceph-mon[23164]: Upgrade: Updating mgr.x 2024-05-06T12:35:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:16 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:16 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:35:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:16 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:35:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:16 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[27137]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[27137]: Upgrade: Updating mgr.x 2024-05-06T12:35:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:35:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:35:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[22413]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[22413]: Upgrade: Updating mgr.x 2024-05-06T12:35:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:35:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:35:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:16 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:17.215 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:17 smithi086 ceph-mon[23164]: Deploying daemon mgr.x on smithi086 2024-05-06T12:35:17.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:17 smithi067 ceph-mon[27137]: Deploying daemon mgr.x on smithi086 2024-05-06T12:35:17.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:17 smithi067 ceph-mon[22413]: Deploying daemon mgr.x on smithi086 2024-05-06T12:35:17.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:17 smithi086 systemd[1]: Stopping Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:35:17.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:17 smithi086 bash[44184]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr.x" found: no such container 2024-05-06T12:35:17.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:17 smithi086 podman[44197]: 2024-05-06 12:35:17.379500692 +0000 UTC m=+0.129806557 container died baa64463db03ac578c2eaa171ece5765d1ceca35d80022d8b8ac991a50812dc3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, release=754, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8) 2024-05-06T12:35:18.038 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:17.779Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:35:18.039 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:17 smithi086 podman[44197]: 2024-05-06 12:35:17.766274145 +0000 UTC m=+0.516579950 container cleanup baa64463db03ac578c2eaa171ece5765d1ceca35d80022d8b8ac991a50812dc3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, architecture=x86_64, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, RELEASE=HEAD, distribution-scope=public) 2024-05-06T12:35:18.039 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:17 smithi086 bash[44197]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x 2024-05-06T12:35:18.039 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:17 smithi086 bash[44224]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr.x" found: no such container 2024-05-06T12:35:18.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:18 smithi086 ceph-mon[23164]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:18.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 podman[44211]: 2024-05-06 12:35:18.043272124 +0000 UTC m=+0.658597477 container remove baa64463db03ac578c2eaa171ece5765d1ceca35d80022d8b8ac991a50812dc3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, io.buildah.version=1.19.8, vendor=Red Hat, Inc., ceph=True, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, maintainer=Guillaume Abrioux , version=8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_BRANCH=HEAD) 2024-05-06T12:35:18.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:35:18.370 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-06T12:35:18.370 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 systemd[1]: Stopped Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:35:18.370 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Consumed 16.857s CPU time. 2024-05-06T12:35:18.370 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 systemd[1]: Starting Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:35:18.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:18 smithi067 ceph-mon[22413]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:18.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:18 smithi067 ceph-mon[27137]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:18.797 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 podman[44334]: 2024-05-06 12:35:18.379082625 +0000 UTC m=+0.017713139 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:35:18.797 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 podman[44334]: 2024-05-06 12:35:18.477741476 +0000 UTC m=+0.116371979 container create 68b03f3b238a970af274dd2e3c83132490d2915b4ac489f3b213280933163990 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-06T12:35:18.797 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 podman[44334]: 2024-05-06 12:35:18.697076889 +0000 UTC m=+0.335707393 container init 68b03f3b238a970af274dd2e3c83132490d2915b4ac489f3b213280933163990 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-06T12:35:18.797 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 podman[44334]: 2024-05-06 12:35:18.700012038 +0000 UTC m=+0.338642542 container start 68b03f3b238a970af274dd2e3c83132490d2915b4ac489f3b213280933163990 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-06T12:35:18.797 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 ceph-mgr[44353]: -- 172.21.15.86:0/1775321435 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d88e9aa5a0 con 0x55d88e97ec00 2024-05-06T12:35:19.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 bash[44334]: 68b03f3b238a970af274dd2e3c83132490d2915b4ac489f3b213280933163990 2024-05-06T12:35:19.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 systemd[1]: Started Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:35:19.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:18.848+0000 7fd445bec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:35:19.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:19.040+0000 7fd445bec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:35:19.480 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:19.157+0000 7fd445bec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:35:19.480 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:19.478+0000 7fd445bec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:35:19.850 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:19.566+0000 7fd445bec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:35:20.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:19 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:20.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:19 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:20.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:19 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:35:20.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:19 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:20.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:20.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:20.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:35:20.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:20.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:20.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:20.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:35:20.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:19 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:20.851 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:20] "GET /metrics HTTP/1.1" 200 214781 "" "Prometheus/2.33.4" 2024-05-06T12:35:21.080 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:20.826+0000 7fd445bec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:35:21.080 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:21.013+0000 7fd445bec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:35:21.080 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:20 smithi086 ceph-mon[23164]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:21.080 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:20 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:21.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:20 smithi067 ceph-mon[27137]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:21.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:20 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:21.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:20 smithi067 ceph-mon[22413]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:21.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:20 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:21.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:21.194+0000 7fd445bec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:35:21.821 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:21.502+0000 7fd445bec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:35:21.822 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:21.606+0000 7fd445bec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:35:21.822 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:21.820+0000 7fd445bec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:35:22.115 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:22.023+0000 7fd445bec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:35:22.853 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:22.655+0000 7fd445bec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:35:23.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:22 smithi086 ceph-mon[23164]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:23.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:23.096+0000 7fd445bec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:35:23.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:22 smithi067 ceph-mon[22413]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:23.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:22 smithi067 ceph-mon[27137]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:23.402 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:23.198+0000 7fd445bec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:35:23.679 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:23.415+0000 7fd445bec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T12:35:23.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:35:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:23.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:35:23.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:24.263 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:23.973+0000 7fd445bec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:35:24.263 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:24.067+0000 7fd445bec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:35:24.263 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:24 smithi086 ceph-mon[23164]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:24.263 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:24 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:24.263 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:24 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:24.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:24 smithi067 ceph-mon[22413]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:24.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:24 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:24.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:24 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:24.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:24 smithi067 ceph-mon[27137]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:24.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:24 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:24.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:24 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:24.552 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:24.320+0000 7fd445bec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:35:24.552 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:24.455+0000 7fd445bec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:35:24.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:24.550+0000 7fd445bec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:35:24.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:24.668+0000 7fd445bec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:35:24.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:24.761+0000 7fd445bec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:35:25.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:25.590+0000 7fd445bec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:35:25.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:25.818+0000 7fd445bec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:35:25.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:25] ENGINE Bus STARTING 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:26 smithi086 ceph-mon[23164]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:26 smithi086 ceph-mon[23164]: Standby manager daemon x restarted 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:26 smithi086 ceph-mon[23164]: Standby manager daemon x started 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:26 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:26 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:26 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:26 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: CherryPy Checker: 2024-05-06T12:35:26.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: The Application mounted at '' has an empty config. 2024-05-06T12:35:26.349 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:35:26.349 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:25] ENGINE Serving on http://:::9283 2024-05-06T12:35:26.349 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:25] ENGINE Bus STARTED 2024-05-06T12:35:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[22413]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[22413]: Standby manager daemon x restarted 2024-05-06T12:35:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[22413]: Standby manager daemon x started 2024-05-06T12:35:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:35:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:35:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:35:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:35:26.451 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[27137]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:26.451 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[27137]: Standby manager daemon x restarted 2024-05-06T12:35:26.451 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[27137]: Standby manager daemon x started 2024-05-06T12:35:26.451 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:35:26.451 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:35:26.451 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:35:26.451 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:26 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:35:26.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:26 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:26.449Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:35:27.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: mgrmap e22: y(active, since 5m), standbys: x 2024-05-06T12:35:27.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:27.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:27.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:35:27.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-06T12:35:27.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: Failing over to other MGR 2024-05-06T12:35:27.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-05-06T12:35:27.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:27 smithi086 ceph-mon[23164]: osdmap e96: 8 total, 8 up, 8 in 2024-05-06T12:35:27.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: mgrmap e22: y(active, since 5m), standbys: x 2024-05-06T12:35:27.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: Failing over to other MGR 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[22413]: osdmap e96: 8 total, 8 up, 8 in 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: mgrmap e22: y(active, since 5m), standbys: x 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-06T12:35:27.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: Failing over to other MGR 2024-05-06T12:35:27.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-05-06T12:35:27.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mon[27137]: osdmap e96: 8 total, 8 up, 8 in 2024-05-06T12:35:28.028 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setuser ceph since I am not root 2024-05-06T12:35:28.028 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ignoring --setgroup ceph since I am not root 2024-05-06T12:35:28.028 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:27 smithi067 ceph-mgr[22649]: -- 172.21.15.67:0/3814944966 <== mon.1 v2:172.21.15.67:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cbb6b7c340 con 0x55cbb6b55000 2024-05-06T12:35:28.028 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:27.891+0000 7fe2f9182000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:35:28.028 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:28.026+0000 7fe2f9182000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:35:28.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:27] ENGINE Bus STOPPING 2024-05-06T12:35:28.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:35:28.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:27] ENGINE Bus STOPPED 2024-05-06T12:35:28.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:27] ENGINE Bus STARTING 2024-05-06T12:35:28.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:27] ENGINE Serving on http://:::9283 2024-05-06T12:35:28.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:35:27] ENGINE Bus STARTED 2024-05-06T12:35:28.463 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:28.130+0000 7fe2f9182000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:35:28.717 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:28.462+0000 7fe2f9182000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:35:28.717 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:28.548+0000 7fe2f9182000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:35:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-05-06T12:35:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: mgrmap e23: x(active, starting, since 0.995915s) 2024-05-06T12:35:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:35:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:35:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:35:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:35:28.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: Manager daemon x is now available 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: mgrmap e23: x(active, starting, since 0.995915s) 2024-05-06T12:35:28.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:35:28.979 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: Manager daemon x is now available 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.980 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:28 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:28.980 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:28.799Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:35:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24343 172.21.15.67:0/297892186' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-05-06T12:35:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: mgrmap e23: x(active, starting, since 0.995915s) 2024-05-06T12:35:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:35:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:35:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:35:29.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:35:29.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:35:29.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:35:29.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:35:29.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:35:29.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:35:29.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:35:29.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:35:29.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:35:29.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:35:29.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:35:29.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:35:29.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: Manager daemon x is now available 2024-05-06T12:35:29.123 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:29.123 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:29.123 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:29.123 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:29.123 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:35:29.123 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:35:29.124 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T12:35:29.124 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T12:35:29.124 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:29.124 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:28 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:29.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:29 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:29.752+0000 7fe2f9182000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:35:29.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:29 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:29.927+0000 7fe2f9182000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:35:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: Queued rgw.foo for migration 2024-05-06T12:35:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: Queued rgw.smpl for migration 2024-05-06T12:35:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-06T12:35:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-05-06T12:35:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: Deploying cephadm binary to smithi086 2024-05-06T12:35:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: Deploying cephadm binary to smithi067 2024-05-06T12:35:29.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: mgrmap e24: x(active, since 1.99708s) 2024-05-06T12:35:29.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[27137]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:29.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: Queued rgw.foo for migration 2024-05-06T12:35:29.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: Queued rgw.smpl for migration 2024-05-06T12:35:29.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-06T12:35:29.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-05-06T12:35:29.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: Deploying cephadm binary to smithi086 2024-05-06T12:35:29.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: Deploying cephadm binary to smithi067 2024-05-06T12:35:29.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: mgrmap e24: x(active, since 1.99708s) 2024-05-06T12:35:29.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:29 smithi067 ceph-mon[22413]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: Queued rgw.foo for migration 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: Queued rgw.smpl for migration 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: Deploying cephadm binary to smithi086 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: Deploying cephadm binary to smithi067 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: mgrmap e24: x(active, since 1.99708s) 2024-05-06T12:35:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:29 smithi086 ceph-mon[23164]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:30.345 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:30.013+0000 7fe2f9182000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:35:30.722 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:30.343+0000 7fe2f9182000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:35:30.723 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:30.441+0000 7fe2f9182000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:35:30.723 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:30.539+0000 7fe2f9182000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:35:30.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:30 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:30.793+0000 7fe2f9182000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:35:30.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[27137]: [06/May/2024:12:35:29] ENGINE Bus STARTING 2024-05-06T12:35:30.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[27137]: [06/May/2024:12:35:29] ENGINE Serving on https://172.21.15.86:7150 2024-05-06T12:35:30.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[27137]: [06/May/2024:12:35:29] ENGINE Serving on http://172.21.15.86:8765 2024-05-06T12:35:30.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[27137]: [06/May/2024:12:35:29] ENGINE Bus STARTED 2024-05-06T12:35:30.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[27137]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:30.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[22413]: [06/May/2024:12:35:29] ENGINE Bus STARTING 2024-05-06T12:35:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[22413]: [06/May/2024:12:35:29] ENGINE Serving on https://172.21.15.86:7150 2024-05-06T12:35:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[22413]: [06/May/2024:12:35:29] ENGINE Serving on http://172.21.15.86:8765 2024-05-06T12:35:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[22413]: [06/May/2024:12:35:29] ENGINE Bus STARTED 2024-05-06T12:35:30.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:30 smithi067 ceph-mon[22413]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:30 smithi086 ceph-mon[23164]: [06/May/2024:12:35:29] ENGINE Bus STARTING 2024-05-06T12:35:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:30 smithi086 ceph-mon[23164]: [06/May/2024:12:35:29] ENGINE Serving on https://172.21.15.86:7150 2024-05-06T12:35:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:30 smithi086 ceph-mon[23164]: [06/May/2024:12:35:29] ENGINE Serving on http://172.21.15.86:8765 2024-05-06T12:35:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:30 smithi086 ceph-mon[23164]: [06/May/2024:12:35:29] ENGINE Bus STARTED 2024-05-06T12:35:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:30 smithi086 ceph-mon[23164]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:31.724 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:31 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:31.265+0000 7fe2f9182000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:35:31.732 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:31] "GET /metrics HTTP/1.1" 200 35140 "" "Prometheus/2.33.4" 2024-05-06T12:35:32.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:31 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:32.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:31 smithi086 ceph-mon[23164]: mgrmap e25: x(active, since 4s) 2024-05-06T12:35:32.195 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:31 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:32.195 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:31 smithi067 ceph-mon[27137]: mgrmap e25: x(active, since 4s) 2024-05-06T12:35:32.195 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:31 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:32.195 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:31 smithi067 ceph-mon[22413]: mgrmap e25: x(active, since 4s) 2024-05-06T12:35:32.195 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:31 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:31.780+0000 7fe2f9182000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:35:32.195 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:31 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:31.879+0000 7fe2f9182000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:35:32.474 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:32.193+0000 7fe2f9182000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:35:32.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:32.288+0000 7fe2f9182000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:35:32.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:32 smithi067 ceph-mon[27137]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:32.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:32 smithi067 ceph-mon[22413]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:32.715+0000 7fe2f9182000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:35:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:32.859+0000 7fe2f9182000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:35:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:32.964+0000 7fe2f9182000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:35:33.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:32 smithi086 ceph-mon[23164]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:35:33.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:33.089+0000 7fe2f9182000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:35:33.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:33.186+0000 7fe2f9182000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:35:33.796 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:33 smithi067 ceph-mon[27137]: mgrmap e26: x(active, since 6s) 2024-05-06T12:35:33.796 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:33 smithi067 ceph-mon[27137]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-05-06T12:35:33.796 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:33 smithi067 ceph-mon[22413]: mgrmap e26: x(active, since 6s) 2024-05-06T12:35:33.796 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:33 smithi067 ceph-mon[22413]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-05-06T12:35:33.796 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:35:33.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:35:33.796 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:35:33.796 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:35:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:33 smithi086 ceph-mon[23164]: mgrmap e26: x(active, since 6s) 2024-05-06T12:35:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:33 smithi086 ceph-mon[23164]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-05-06T12:35:34.175 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:33.794+0000 7fe2f9182000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:35:34.475 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:34.255Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:34.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:34.173+0000 7fe2f9182000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:35:34.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:35:34] ENGINE Bus STARTING 2024-05-06T12:35:34.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: CherryPy Checker: 2024-05-06T12:35:34.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: The Application mounted at '' has an empty config. 2024-05-06T12:35:34.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: 2024-05-06T12:35:34.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:35:34] ENGINE Serving on http://:::9283 2024-05-06T12:35:34.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: [06/May/2024:12:35:34] ENGINE Bus STARTED 2024-05-06T12:35:34.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[22413]: Standby manager daemon y started 2024-05-06T12:35:34.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T12:35:34.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:35:34.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T12:35:34.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:35:34.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[27137]: Standby manager daemon y started 2024-05-06T12:35:34.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T12:35:34.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:35:34.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T12:35:34.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:34 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:35:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:34 smithi086 ceph-mon[23164]: Standby manager daemon y started 2024-05-06T12:35:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:34 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T12:35:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:34 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:35:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:34 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T12:35:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:34 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/2460817629' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:35:36.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:35 smithi086 ceph-mon[23164]: mgrmap e27: x(active, since 8s), standbys: y 2024-05-06T12:35:36.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:35 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:35:36.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:35 smithi086 ceph-mon[23164]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T12:35:36.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:35 smithi067 ceph-mon[22413]: mgrmap e27: x(active, since 8s), standbys: y 2024-05-06T12:35:36.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:35 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:35:36.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:35 smithi067 ceph-mon[22413]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T12:35:36.194 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:35 smithi067 ceph-mon[27137]: mgrmap e27: x(active, since 8s), standbys: y 2024-05-06T12:35:36.194 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:35 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:35:36.194 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:35 smithi067 ceph-mon[27137]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T12:35:38.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:37 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:38.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:37 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:38.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:37 smithi086 ceph-mon[23164]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T12:35:38.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:37 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:38.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:37 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:38.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:37 smithi067 ceph-mon[22413]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T12:35:38.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:37 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:38.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:37 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:38.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:37 smithi067 ceph-mon[27137]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T12:35:38.949 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:35:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi067 to 2736M 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi067 to 2736M 2024-05-06T12:35:39.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:38 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.310 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi067 to 2736M 2024-05-06T12:35:39.311 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:38 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:39.622 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (4m) 2s ago 4m 27.4M - 0.23.0 ba2b418f427c f43d96904837 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (4m) 16s ago 4m 55.6M - 8.3.5 dad864ee21e9 87f103efe219 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (3m) 2s ago 3m 51.0M - 3.5 e1d6a67b021e 1a04cb514a79 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283 running (21s) 16s ago 9m 41.6M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 68b03f3b238a 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:9283 running (10m) 2s ago 10m 371M - 17.2.0 e1d6a67b021e 156e546a6ad8 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (11m) 2s ago 11m 51.3M 2048M 17.2.0 e1d6a67b021e f5b02626cc9d 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (9m) 16s ago 9m 40.0M 2048M 17.2.0 e1d6a67b021e 5cb539dec1ee 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (9m) 2s ago 9m 41.5M 2048M 17.2.0 e1d6a67b021e d0561ab1fc3c 2024-05-06T12:35:39.623 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (5m) 2s ago 5m 17.2M - 1.3.1 1dbe0e931976 50bfd17cec65 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (5m) 16s ago 5m 14.3M - 1dbe0e931976 b88ef5b0017c 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (9m) 2s ago 9m 50.9M 2736M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (8m) 2s ago 8m 52.3M 2736M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (8m) 2s ago 8m 46.0M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (7m) 2s ago 7m 49.9M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (7m) 16s ago 7m 50.9M 4272M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (7m) 16s ago 7m 47.7M 4272M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:35:39.624 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (6m) 16s ago 6m 49.9M 4272M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:35:39.625 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (6m) 16s ago 6m 51.9M 4272M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:35:39.625 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (4m) 16s ago 5m 66.9M - 514e6a882f6e af867ca274de 2024-05-06T12:35:39.625 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (3m) 2s ago 3m 72.7M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:35:39.625 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (3m) 16s ago 3m 70.3M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:35:39.625 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (3m) 2s ago 3m 72.7M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:35:39.625 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (3m) 16s ago 3m 70.0M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:35:40.034 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": {}, 2024-05-06T12:35:40.035 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:35:40.036 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:35:40.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:35:40.036 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:35:40.036 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-05-06T12:35:40.036 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T12:35:40.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:35:40.036 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[22413]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[22413]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[22413]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[22413]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[27137]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[27137]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[27137]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:39 smithi067 ceph-mon[27137]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:35:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:39 smithi086 ceph-mon[23164]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:39 smithi086 ceph-mon[23164]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:39 smithi086 ceph-mon[23164]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:40.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:39 smithi086 ceph-mon[23164]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:35:40.378 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:35:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:35:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:35:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:35:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-05-06T12:35:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "1/23 daemons upgraded", 2024-05-06T12:35:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-05-06T12:35:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:35:40.379 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:35:40.724 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:40 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:35:40.789 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/487534250' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[22413]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/14417767' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/487534250' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[27137]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:41 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/14417767' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:35:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:41 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:41 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/487534250' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:35:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:41 smithi086 ceph-mon[23164]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:35:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:41 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/14417767' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:35:41.786 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:41] "GET /metrics HTTP/1.1" 200 35140 "" "Prometheus/2.33.4" 2024-05-06T12:35:42.315 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:42 smithi086 ceph-mon[23164]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:35:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:42 smithi067 ceph-mon[22413]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:35:42.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:42 smithi067 ceph-mon[27137]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:35:42.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:35:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:35:42.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.86:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.67:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:35:43.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:43 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:43.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:43 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:43 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:43.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:35:43.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:35:43.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:43.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:35:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:44 smithi086 ceph-mon[23164]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:35:44.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:44 smithi067 ceph-mon[27137]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:35:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:44 smithi067 ceph-mon[22413]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:35:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:46 smithi086 ceph-mon[23164]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:46.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:46 smithi067 ceph-mon[22413]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:46.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:46 smithi067 ceph-mon[27137]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:47 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:47.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:47 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:47 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:47.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:47 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:47 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:47 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:48 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:48 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:48 smithi067 ceph-mon[22413]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:48 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:48 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:48 smithi067 ceph-mon[27137]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:48.535 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:48 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:48.535 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:48 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:48.535 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:48 smithi086 ceph-mon[23164]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:49 smithi086 ceph-mon[23164]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:50.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:49 smithi067 ceph-mon[22413]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:50.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:49 smithi067 ceph-mon[27137]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:50.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:35:50 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:35:51.761 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:35:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ::ffff:172.21.15.86 - - [06/May/2024:12:35:51] "GET /metrics HTTP/1.1" 200 38137 "" "Prometheus/2.33.4" 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.762 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: Adjusting osd_memory_target on smithi086 to 2992M 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:51 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: Adjusting osd_memory_target on smithi086 to 2992M 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:35:51.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T12:35:51.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: Adjusting osd_memory_target on smithi086 to 2992M 2024-05-06T12:35:51.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:35:51.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:35:51.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:35:51.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:35:51.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:51.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:51 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:35:52 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:35:52.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.86:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.67:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:35:52.634 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:52 smithi067 systemd[1]: Stopping Ceph node-exporter.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:35:52.634 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:52 smithi067 bash[56154]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter.a" found: no such container 2024-05-06T12:35:52.634 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:52 smithi067 podman[56167]: 2024-05-06 12:35:52.386099988 +0000 UTC m=+0.120755110 container died 50bfd17cec6591f6f4f7ca4af80e14c6533107f260346c025bc38369314cc2c2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:52.888 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:52 smithi067 podman[56167]: 2024-05-06 12:35:52.85073585 +0000 UTC m=+0.585390976 container cleanup 50bfd17cec6591f6f4f7ca4af80e14c6533107f260346c025bc38369314cc2c2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:52.888 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:52 smithi067 bash[56167]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a 2024-05-06T12:35:52.888 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:52 smithi067 bash[56193]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter.a" found: no such container 2024-05-06T12:35:52.888 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:35:52.888 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:35:52.888 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:35:52.888 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[22413]: Reconfiguring daemon node-exporter.a on smithi067 2024-05-06T12:35:52.889 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-06T12:35:52.890 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:52 smithi067 ceph-mon[27137]: Reconfiguring daemon node-exporter.a on smithi067 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:53.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:53.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-06T12:35:53.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:52 smithi086 ceph-mon[23164]: Reconfiguring daemon node-exporter.a on smithi067 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 podman[56181]: 2024-05-06 12:35:53.110921211 +0000 UTC m=+0.728929102 container remove 50bfd17cec6591f6f4f7ca4af80e14c6533107f260346c025bc38369314cc2c2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 systemd[1]: Stopped Ceph node-exporter.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.a.service: Consumed 2.895s CPU time. 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 systemd[1]: Starting Ceph node-exporter.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 podman[56262]: 2024-05-06 12:35:53.244141638 +0000 UTC m=+0.018532102 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T12:35:53.475 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 podman[56262]: 2024-05-06 12:35:53.353400624 +0000 UTC m=+0.127791086 container create 347a17f8d7d6dbe88c2b46790dcc4daa85ca2795f133c6d2bedb2778546b7aed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:53.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=error ts=2024-05-06T12:35:53.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:53.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.67:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.67 because it doesn't contain any IP SANs" 2024-05-06T12:35:53.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=warn ts=2024-05-06T12:35:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.86:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.86 because it doesn't contain any IP SANs" 2024-05-06T12:35:53.975 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 podman[56262]: 2024-05-06 12:35:53.589188464 +0000 UTC m=+0.363578925 container init 347a17f8d7d6dbe88c2b46790dcc4daa85ca2795f133c6d2bedb2778546b7aed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:53.975 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 podman[56262]: 2024-05-06 12:35:53.592167382 +0000 UTC m=+0.366557847 container start 347a17f8d7d6dbe88c2b46790dcc4daa85ca2795f133c6d2bedb2778546b7aed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:53.975 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.595Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T12:35:53.975 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.595Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.595Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.595Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T12:35:53.976 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T12:35:53.977 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T12:35:53.978 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T12:35:53.979 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T12:35:53.979 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T12:35:53.979 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T12:35:53.979 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T12:35:53.979 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T12:35:53.979 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.596Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T12:35:53.980 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.597Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T12:35:53.981 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a[56277]: ts=2024-05-06T12:35:53.597Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T12:35:53.981 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 bash[56262]: 347a17f8d7d6dbe88c2b46790dcc4daa85ca2795f133c6d2bedb2778546b7aed 2024-05-06T12:35:53.981 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:35:53 smithi067 systemd[1]: Started Ceph node-exporter.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:35:54.912 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:54 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:54.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:54 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:54.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:54 smithi067 ceph-mon[22413]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:54.913 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:54 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:54.913 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:54 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:54.913 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:54 smithi067 ceph-mon[27137]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:55.124 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:54 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:55.125 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:54 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:35:55.125 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:54 smithi086 ceph-mon[23164]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:55.862 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:55 smithi067 ceph-mon[22413]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T12:35:55.862 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:55 smithi067 ceph-mon[22413]: Reconfiguring daemon alertmanager.a on smithi067 2024-05-06T12:35:55.862 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:55 smithi067 ceph-mon[27137]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T12:35:55.862 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:55 smithi067 ceph-mon[27137]: Reconfiguring daemon alertmanager.a on smithi067 2024-05-06T12:35:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:55 smithi086 ceph-mon[23164]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T12:35:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:55 smithi086 ceph-mon[23164]: Reconfiguring daemon alertmanager.a on smithi067 2024-05-06T12:35:56.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:56 smithi067 ceph-mon[22413]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:56.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:56 smithi067 ceph-mon[27137]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:57.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:56 smithi086 ceph-mon[23164]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:35:57.431 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:57 smithi067 systemd[1]: Stopping Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:35:57.431 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:57 smithi067 bash[56462]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T12:35:57.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:57 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[47150]: level=info ts=2024-05-06T12:35:57.430Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-06T12:35:57.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:57 smithi067 podman[56475]: 2024-05-06 12:35:57.552709295 +0000 UTC m=+0.286484059 container died f43d96904837b660b5b06d2a9111ac8534d05a2445c2b8b83a4d28b44ebe9df1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:58.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:57 smithi086 ceph-mon[23164]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:58.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:57 smithi067 ceph-mon[27137]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:57 smithi067 ceph-mon[22413]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:35:58.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:57 smithi067 podman[56475]: 2024-05-06 12:35:57.928002671 +0000 UTC m=+0.661777434 container cleanup f43d96904837b660b5b06d2a9111ac8534d05a2445c2b8b83a4d28b44ebe9df1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:58.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:57 smithi067 bash[56475]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a 2024-05-06T12:35:58.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:57 smithi067 bash[56504]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T12:35:58.589 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 podman[56489]: 2024-05-06 12:35:58.250747303 +0000 UTC m=+0.697417378 container remove f43d96904837b660b5b06d2a9111ac8534d05a2445c2b8b83a4d28b44ebe9df1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:58.590 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 podman[56489]: 2024-05-06 12:35:58.338649168 +0000 UTC m=+0.785319243 volume remove bf1bff2f461a7ac04e571e84dddb318bcf5ef8f7df0381d93a3c62f6f69d9b2c 2024-05-06T12:35:58.590 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-06T12:35:58.590 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 systemd[1]: Stopped Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:35:58.590 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 systemd[1]: Starting Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:35:58.887 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:35:58 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:58.888 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:35:58 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:58.888 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 podman[56576]: 2024-05-06 12:35:58.493040591 +0000 UTC m=+0.019331609 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-05-06T12:35:58.888 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 podman[56576]: 2024-05-06 12:35:58.606390228 +0000 UTC m=+0.132681288 volume create b0ba3f6a87e893c619eff3e33f264a625e7cbb3924506464ffa09bf0bb34de46 2024-05-06T12:35:58.888 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:58 smithi067 podman[56576]: 2024-05-06 12:35:58.698559891 +0000 UTC m=+0.224850907 container create 130b40fe12565a875a0106f759b28421cbadeb4882dfdfeef5bed0f03ca40869 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:59.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:35:58 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:35:59.225 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 podman[56576]: 2024-05-06 12:35:59.009623944 +0000 UTC m=+0.535914962 container init 130b40fe12565a875a0106f759b28421cbadeb4882dfdfeef5bed0f03ca40869 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:59.225 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 podman[56576]: 2024-05-06 12:35:59.012391843 +0000 UTC m=+0.538682860 container start 130b40fe12565a875a0106f759b28421cbadeb4882dfdfeef5bed0f03ca40869 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:35:59.225 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.022Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-06T12:35:59.225 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.023Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.023Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.67 port=9094 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.024Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.050Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.051Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.053Z caller=main.go:518 msg=Listening address=:9093 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:35:59.053Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 bash[56576]: 130b40fe12565a875a0106f759b28421cbadeb4882dfdfeef5bed0f03ca40869 2024-05-06T12:35:59.226 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:35:59 smithi067 systemd[1]: Started Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:00.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: Reconfiguring iscsi.foo.smithi067.fnpkny (dependencies changed)... 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: Reconfiguring daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[22413]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:00.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: Reconfiguring iscsi.foo.smithi067.fnpkny (dependencies changed)... 2024-05-06T12:36:00.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:00.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:00.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:00.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: Reconfiguring daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:36:00.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:00 smithi067 ceph-mon[27137]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: Reconfiguring iscsi.foo.smithi067.fnpkny (dependencies changed)... 2024-05-06T12:36:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:00.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:00.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: Reconfiguring daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:36:00.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:00 smithi086 ceph-mon[23164]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:00.860 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:00 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[22645]: ::ffff:172.21.15.86 - - [06/May/2024:12:36:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:36:01.122 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:36:01 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:36:01.025Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00030558s 2024-05-06T12:36:01.423 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:01 smithi086 ceph-mon[23164]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:01.439 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:01 smithi067 ceph-mon[22413]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:01.439 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:01 smithi067 ceph-mon[27137]: from='client.14847 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:01.833 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:01 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ::ffff:172.21.15.86 - - [06/May/2024:12:36:01] "GET /metrics HTTP/1.1" 200 38135 "" "Prometheus/2.33.4" 2024-05-06T12:36:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:02 smithi067 ceph-mon[22413]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:02.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:02 smithi067 ceph-mon[27137]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:02 smithi086 ceph-mon[23164]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:02.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:02.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.86:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.67:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:36:03.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:02.715+0000 7fd3cb7b4700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Creating ceph-iscsi config... 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Traceback (most recent call last): 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: exec(code, run_globals) 2024-05-06T12:36:03.121 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.121 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.121 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.121 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.121 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.121 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.121 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Traceback (most recent call last): 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return self.event_loop.get_result(coro, timeout) 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return future.result(timeout) 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T12:36:03.122 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return self.__get_result() 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: raise self._exception 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: use_current_daemon_image=reconfig, 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Creating ceph-iscsi config... 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.123 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Traceback (most recent call last): 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: exec(code, run_globals) 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.124 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:03 smithi067 ceph-mon[22413]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: Creating ceph-iscsi config... 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: exec(code, run_globals) 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T12:36:03.477 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: return future.result(timeout) 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: return self.__get_result() 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: raise self._exception 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: use_current_daemon_image=reconfig, 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: Creating ceph-iscsi config... 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.478 INFO:journalctl@ceph.mon.a.smithi067.stdout: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: exec(code, run_globals) 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.479 INFO:journalctl@ceph.mon.a.smithi067.stdout: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.a.smithi067.stdout: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:03 smithi067 ceph-mon[22413]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:03 smithi067 ceph-mon[27137]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.c.smithi067.stdout: Creating ceph-iscsi config... 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.c.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.c.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.c.smithi067.stdout: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.c.smithi067.stdout: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.480 INFO:journalctl@ceph.mon.c.smithi067.stdout: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: exec(code, run_globals) 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.481 INFO:journalctl@ceph.mon.c.smithi067.stdout: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: return future.result(timeout) 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: return self.__get_result() 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: raise self._exception 2024-05-06T12:36:03.482 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: use_current_daemon_image=reconfig, 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: Creating ceph-iscsi config... 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: exec(code, run_globals) 2024-05-06T12:36:03.483 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.484 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:03 smithi067 ceph-mon[27137]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:36:03.487 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:03 smithi086 ceph-mon[23164]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.487 INFO:journalctl@ceph.mon.b.smithi086.stdout: Creating ceph-iscsi config... 2024-05-06T12:36:03.487 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.487 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: exec(code, run_globals) 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.488 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: return future.result(timeout) 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: return self.__get_result() 2024-05-06T12:36:03.489 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: raise self._exception 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: use_current_daemon_image=reconfig, 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: Creating ceph-iscsi config... 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.490 INFO:journalctl@ceph.mon.b.smithi086.stdout: Traceback (most recent call last): 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: return _run_code(code, main_globals, None, 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: exec(code, run_globals) 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:03.491 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:03 smithi086 ceph-mon[23164]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:36:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:04 smithi067 ceph-mon[22413]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:36:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:04 smithi067 ceph-mon[22413]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:04 smithi067 ceph-mon[27137]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:36:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:04 smithi067 ceph-mon[27137]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:04.602 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:04 smithi086 ceph-mon[23164]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:36:04.603 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:04 smithi086 ceph-mon[23164]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:06.210 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:06 smithi086 ceph-mon[23164]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:06.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:06 smithi067 ceph-mon[22413]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:06 smithi067 ceph-mon[27137]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:06.523 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 systemd[1]: Stopping Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:06.524 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 bash[47710]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T12:36:06.815 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.522Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T12:36:06.815 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T12:36:06.815 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T12:36:06.815 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T12:36:06.815 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[40054]: ts=2024-05-06T12:36:06.523Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T12:36:06.816 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 podman[47722]: 2024-05-06 12:36:06.53315494 +0000 UTC m=+0.149903007 container died af867ca274deb443ff21d77543dfa1aa536f13f87938130ef2a0516496b2705c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:07.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 podman[47722]: 2024-05-06 12:36:06.921932527 +0000 UTC m=+0.538680590 container cleanup af867ca274deb443ff21d77543dfa1aa536f13f87938130ef2a0516496b2705c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:07.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 bash[47722]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T12:36:07.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:06 smithi086 bash[47749]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T12:36:07.466 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 podman[47736]: 2024-05-06 12:36:07.172960311 +0000 UTC m=+0.641980656 container remove af867ca274deb443ff21d77543dfa1aa536f13f87938130ef2a0516496b2705c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:07.466 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T12:36:07.466 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 systemd[1]: Stopped Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:07.466 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@prometheus.a.service: Consumed 1.487s CPU time. 2024-05-06T12:36:07.467 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 systemd[1]: Starting Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:07.467 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 podman[47818]: 2024-05-06 12:36:07.311201973 +0000 UTC m=+0.019650411 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T12:36:07.467 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 podman[47818]: 2024-05-06 12:36:07.465814026 +0000 UTC m=+0.174262507 container create 92894a1d8e345e98d6b45512c32952a24b0faee7f43800c0b45d88263273c7e3 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:07.743 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 podman[47818]: 2024-05-06 12:36:07.667991866 +0000 UTC m=+0.376440305 container init 92894a1d8e345e98d6b45512c32952a24b0faee7f43800c0b45d88263273c7e3 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:07.743 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 podman[47818]: 2024-05-06 12:36:07.670506955 +0000 UTC m=+0.378955394 container start 92894a1d8e345e98d6b45512c32952a24b0faee7f43800c0b45d88263273c7e3 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:07.743 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.693Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.693Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.693Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.693Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi086 (none))" 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.693Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.693Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.695Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.695Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.696Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.698Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.698Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.6µs 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.698Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T12:36:07.744 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 bash[47818]: 92894a1d8e345e98d6b45512c32952a24b0faee7f43800c0b45d88263273c7e3 2024-05-06T12:36:08.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:07 smithi086 ceph-mon[23164]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:36:08.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:07 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:08.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:07 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:08.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 systemd[1]: Started Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:08.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:07.901Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-05-06T12:36:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:07 smithi067 ceph-mon[22413]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:36:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:07 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:07 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:08.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:07 smithi067 ceph-mon[27137]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:36:08.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:07 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:08.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:07 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:08.457 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:08 smithi086 systemd[1]: Stopping Ceph node-exporter.b for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:08.457 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:08 smithi086 bash[47941]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter.b" found: no such container 2024-05-06T12:36:08.457 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.150Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-05-06T12:36:08.457 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.151Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-05-06T12:36:08.457 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.151Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.443µs wal_replay_duration=452.412766ms total_replay_duration=452.454527ms 2024-05-06T12:36:08.457 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.154Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T12:36:08.457 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.154Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T12:36:08.457 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.154Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T12:36:08.457 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.175Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.567995ms db_storage=714ns remote_storage=1.534µs web_handler=435ns query_engine=757ns scrape=374.73µs scrape_sd=110.657µs notify=9.997µs notify_sd=13.484µs rules=20.792891ms 2024-05-06T12:36:08.458 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:08.175Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T12:36:08.764 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:08 smithi086 podman[47951]: 2024-05-06 12:36:08.456215498 +0000 UTC m=+0.104548579 container died b88ef5b0017c7ac6e8a65a9284395727a88ecc99fb35aa9eabda4a7e6b77cff9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:36:09.082 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:08 smithi086 ceph-mon[23164]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-06T12:36:09.083 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:08 smithi086 ceph-mon[23164]: Reconfiguring daemon node-exporter.b on smithi086 2024-05-06T12:36:09.083 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:08 smithi086 podman[47951]: 2024-05-06 12:36:08.854478842 +0000 UTC m=+0.502811934 container cleanup b88ef5b0017c7ac6e8a65a9284395727a88ecc99fb35aa9eabda4a7e6b77cff9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:36:09.083 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:08 smithi086 bash[47951]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b 2024-05-06T12:36:09.083 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:08 smithi086 bash[47978]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter.b" found: no such container 2024-05-06T12:36:09.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:08 smithi067 ceph-mon[22413]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-06T12:36:09.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:08 smithi067 ceph-mon[22413]: Reconfiguring daemon node-exporter.b on smithi086 2024-05-06T12:36:09.163 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:36:09 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:36:09.027Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.0021381s 2024-05-06T12:36:09.164 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:08 smithi067 ceph-mon[27137]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-06T12:36:09.164 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:08 smithi067 ceph-mon[27137]: Reconfiguring daemon node-exporter.b on smithi086 2024-05-06T12:36:09.339 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 podman[47965]: 2024-05-06 12:36:09.105689049 +0000 UTC m=+0.653635244 container remove b88ef5b0017c7ac6e8a65a9284395727a88ecc99fb35aa9eabda4a7e6b77cff9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:36:09.339 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:36:09.339 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-06T12:36:09.339 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 systemd[1]: Stopped Ceph node-exporter.b for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:09.339 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.b.service: Consumed 2.683s CPU time. 2024-05-06T12:36:09.339 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 systemd[1]: Starting Ceph node-exporter.b for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:09.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 podman[48047]: 2024-05-06 12:36:09.241541205 +0000 UTC m=+0.021965530 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T12:36:09.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 podman[48047]: 2024-05-06 12:36:09.340126693 +0000 UTC m=+0.120551005 container create f4564f85d02326d870f3569845c24cc9bf746568f85df6c382c75d1b88a78859 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:36:09.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 podman[48047]: 2024-05-06 12:36:09.509155654 +0000 UTC m=+0.289579974 container init f4564f85d02326d870f3569845c24cc9bf746568f85df6c382c75d1b88a78859 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:36:09.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 podman[48047]: 2024-05-06 12:36:09.512991835 +0000 UTC m=+0.293416147 container start f4564f85d02326d870f3569845c24cc9bf746568f85df6c382c75d1b88a78859 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.515Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.515Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.516Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.516Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.516Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T12:36:09.621 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T12:36:09.622 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T12:36:09.623 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T12:36:09.623 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T12:36:09.623 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T12:36:09.623 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T12:36:09.623 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T12:36:09.623 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T12:36:09.623 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T12:36:09.624 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T12:36:09.624 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T12:36:09.624 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T12:36:09.624 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T12:36:09.624 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T12:36:09.624 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T12:36:09.624 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b[48063]: ts=2024-05-06T12:36:09.517Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T12:36:09.625 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 bash[48047]: f4564f85d02326d870f3569845c24cc9bf746568f85df6c382c75d1b88a78859 2024-05-06T12:36:09.626 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:36:09 smithi086 systemd[1]: Started Ceph node-exporter.b for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:10.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: Reconfiguring grafana.a (dependencies changed)... 2024-05-06T12:36:10.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-06T12:36:10.653 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:36:10.653 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.653 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.653 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:10 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: Reconfiguring grafana.a (dependencies changed)... 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: Reconfiguring grafana.a (dependencies changed)... 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:10.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:10 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:36:11.160 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:36:11.734 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:11 smithi086 ceph-mon[23164]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:36:11.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:11 smithi086 ceph-mon[23164]: Reconfiguring daemon grafana.a on smithi086 2024-05-06T12:36:11.833 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 starting - - - - 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (4m) 25s ago 4m 55.6M - 8.3.5 dad864ee21e9 87f103efe219 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (4m) 35s ago 4m 51.0M - 3.5 e1d6a67b021e 1a04cb514a79 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283 running (53s) 25s ago 9m 491M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 68b03f3b238a 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:9283 running (11m) 35s ago 11m 371M - 17.2.0 e1d6a67b021e 156e546a6ad8 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (11m) 35s ago 11m 51.3M 2048M 17.2.0 e1d6a67b021e f5b02626cc9d 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (10m) 25s ago 10m 43.2M 2048M 17.2.0 e1d6a67b021e 5cb539dec1ee 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (10m) 35s ago 10m 41.5M 2048M 17.2.0 e1d6a67b021e d0561ab1fc3c 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 starting - - - - 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 starting - - - - 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (9m) 35s ago 9m 50.9M 2736M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (9m) 35s ago 9m 52.3M 2736M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (8m) 35s ago 8m 46.0M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (8m) 35s ago 8m 49.9M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (7m) 25s ago 7m 51.4M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (7m) 25s ago 7m 48.2M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (7m) 25s ago 7m 50.5M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (6m) 25s ago 6m 52.6M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 starting - - - - 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (4m) 35s ago 4m 72.7M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (4m) 25s ago 4m 71.0M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (4m) 35s ago 4m 72.7M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (4m) 25s ago 4m 70.9M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:36:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:11 smithi067 ceph-mon[22413]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:36:11.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:11 smithi067 ceph-mon[22413]: Reconfiguring daemon grafana.a on smithi086 2024-05-06T12:36:11.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:11 smithi067 ceph-mon[27137]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T12:36:11.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:11 smithi067 ceph-mon[27137]: Reconfiguring daemon grafana.a on smithi086 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": {}, 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:36:12.245 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "1/23 daemons upgraded", 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-05-06T12:36:12.589 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:36:12.590 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:36:12.895 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:12 smithi086 systemd[1]: Stopping Ceph grafana.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:12.895 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:12 smithi086 bash[48234]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana.a" found: no such container 2024-05-06T12:36:12.895 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:12 smithi086 ceph-mon[23164]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:12.895 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:12 smithi086 ceph-mon[23164]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:12.896 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:12 smithi086 ceph-mon[23164]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T12:36:12.896 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:12 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2243625032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[27137]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[27137]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[27137]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2243625032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[22413]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[22413]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[22413]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T12:36:12.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:12 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2243625032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:36:13.035 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:36:13.212 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[39254]: t=2024-05-06T12:36:12+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-05-06T12:36:13.212 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:12 smithi086 podman[48245]: 2024-05-06 12:36:12.904270572 +0000 UTC m=+0.129355369 container died 87f103efe219c8478872744f5445261cc3fc7d1ebeb0c5172cf1b6a3a8f92a7d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, release=236.1648460182, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, maintainer=Paul Cuzner , name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, version=8.5, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, distribution-scope=public, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vcs-type=git) 2024-05-06T12:36:13.536 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 podman[48245]: 2024-05-06 12:36:13.293421318 +0000 UTC m=+0.518506097 container cleanup 87f103efe219c8478872744f5445261cc3fc7d1ebeb0c5172cf1b6a3a8f92a7d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, version=8.5, io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, architecture=x86_64, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-05-06T12:36:13.536 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 bash[48245]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a 2024-05-06T12:36:13.537 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 bash[48271]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana.a" found: no such container 2024-05-06T12:36:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:13 smithi086 ceph-mon[23164]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:13 smithi086 ceph-mon[23164]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:13 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:13 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2138647754' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:36:13.870 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 podman[48259]: 2024-05-06 12:36:13.536040413 +0000 UTC m=+0.631729484 container remove 87f103efe219c8478872744f5445261cc3fc7d1ebeb0c5172cf1b6a3a8f92a7d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, maintainer=Paul Cuzner , vcs-type=git, vendor=Red Hat, Inc., version=8.5, distribution-scope=public, io.openshift.expose-services=, architecture=x86_64, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.buildah.version=1.24.2, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-06T12:36:13.870 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-06T12:36:13.870 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 systemd[1]: Stopped Ceph grafana.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:13.870 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@grafana.a.service: Consumed 2.555s CPU time. 2024-05-06T12:36:13.870 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 systemd[1]: Starting Ceph grafana.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:13.870 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 podman[48339]: 2024-05-06 12:36:13.666082605 +0000 UTC m=+0.017155377 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-05-06T12:36:13.870 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:13 smithi086 podman[48339]: 2024-05-06 12:36:13.785581942 +0000 UTC m=+0.136654698 container create 3a4e37b2de3aa28e152cb602a8a18aea0993e2f70faa2a781560b5b984f84cc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vcs-type=git, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, version=8.5, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, name=ubi8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-06T12:36:13.933 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[22413]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:13.933 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[22413]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:13.933 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:13.933 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2138647754' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:36:13.933 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[27137]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:13.933 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[27137]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:13.934 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:13.934 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:13 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2138647754' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:36:14.202 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 podman[48339]: 2024-05-06 12:36:14.030031354 +0000 UTC m=+0.381104113 container init 3a4e37b2de3aa28e152cb602a8a18aea0993e2f70faa2a781560b5b984f84cc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, architecture=x86_64, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, release=236.1648460182, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-05-06T12:36:14.202 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 podman[48339]: 2024-05-06 12:36:14.032764684 +0000 UTC m=+0.383837449 container start 3a4e37b2de3aa28e152cb602a8a18aea0993e2f70faa2a781560b5b984f84cc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.component=ubi8-container, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , description=Ceph Grafana Container, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, release=236.1648460182, build-date=2022-03-28T10:36:18.413762) 2024-05-06T12:36:14.202 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-05-06T12:36:14.202 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-05-06T12:36:14.203 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="App mode production" logger=settings 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=513.022µs 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-05-06T12:36:14.204 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-05-06T12:36:14.205 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 bash[48339]: 3a4e37b2de3aa28e152cb602a8a18aea0993e2f70faa2a781560b5b984f84cc1 2024-05-06T12:36:14.205 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 systemd[1]: Started Ceph grafana.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:14.205 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-05-06T12:36:14.205 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-06T12:36:14.205 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-05-06T12:36:14.460 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-05-06T12:36:14.460 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-05-06T12:36:14.460 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:36:14+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-05-06T12:36:14.460 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STOPPING 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STOPPED 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STARTING 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Serving on http://:::9283 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STARTED 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STOPPING 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STOPPED 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STARTING 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Serving on http://:::9283 2024-05-06T12:36:14.461 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STARTED 2024-05-06T12:36:14.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STOPPING 2024-05-06T12:36:14.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:36:14.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STOPPED 2024-05-06T12:36:14.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STARTING 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Serving on http://:::9283 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:14] ENGINE Bus STARTED 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086:9095"}]: dispatch 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:36:14.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086:3000"}]: dispatch 2024-05-06T12:36:14.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:14.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1516707478' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:36:14.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2430627931' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]: dispatch 2024-05-06T12:36:14.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:14 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]: dispatch 2024-05-06T12:36:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T12:36:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086:9095"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086:3000"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1516707478' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2430627931' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]: dispatch 2024-05-06T12:36:14.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086:9095"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086:3000"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:14.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1516707478' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:36:14.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2430627931' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]: dispatch 2024-05-06T12:36:14.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:14 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]: dispatch 2024-05-06T12:36:15.934 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:36:15.934 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T12:36:15.934 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:15.934 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086:9095"}]: dispatch 2024-05-06T12:36:15.934 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:36:15.934 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086:3000"}]: dispatch 2024-05-06T12:36:15.934 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]': finished 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: osdmap e97: 8 total, 8 up, 8 in 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3176091934' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086:9095"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086:3000"}]: dispatch 2024-05-06T12:36:15.935 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T12:36:15.936 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T12:36:15.936 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]': finished 2024-05-06T12:36:15.936 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: osdmap e97: 8 total, 8 up, 8 in 2024-05-06T12:36:15.936 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3176091934' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]: dispatch 2024-05-06T12:36:15.936 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:15 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]: dispatch 2024-05-06T12:36:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T12:36:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T12:36:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086:9095"}]: dispatch 2024-05-06T12:36:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T12:36:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086:3000"}]: dispatch 2024-05-06T12:36:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T12:36:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T12:36:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2089669879"}]': finished 2024-05-06T12:36:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: osdmap e97: 8 total, 8 up, 8 in 2024-05-06T12:36:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3176091934' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]: dispatch 2024-05-06T12:36:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:15 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]: dispatch 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]': finished 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: osdmap e98: 8 total, 8 up, 8 in 2024-05-06T12:36:16.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3101951190' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/976842880"}]: dispatch 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]': finished 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: osdmap e98: 8 total, 8 up, 8 in 2024-05-06T12:36:16.856 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:16 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3101951190' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/976842880"}]: dispatch 2024-05-06T12:36:17.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:17.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:17.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/976842880"}]': finished 2024-05-06T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: osdmap e98: 8 total, 8 up, 8 in 2024-05-06T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:16 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3101951190' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/976842880"}]: dispatch 2024-05-06T12:36:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[22413]: Upgrade: Updating mgr.y 2024-05-06T12:36:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[22413]: Deploying daemon mgr.y on smithi067 2024-05-06T12:36:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3101951190' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/976842880"}]': finished 2024-05-06T12:36:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[22413]: osdmap e99: 8 total, 8 up, 8 in 2024-05-06T12:36:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2443342655' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]: dispatch 2024-05-06T12:36:17.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]: dispatch 2024-05-06T12:36:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[27137]: Upgrade: Updating mgr.y 2024-05-06T12:36:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[27137]: Deploying daemon mgr.y on smithi067 2024-05-06T12:36:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3101951190' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/976842880"}]': finished 2024-05-06T12:36:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[27137]: osdmap e99: 8 total, 8 up, 8 in 2024-05-06T12:36:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2443342655' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]: dispatch 2024-05-06T12:36:17.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:17 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]: dispatch 2024-05-06T12:36:18.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:17 smithi086 ceph-mon[23164]: Upgrade: Updating mgr.y 2024-05-06T12:36:18.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:17 smithi086 ceph-mon[23164]: Deploying daemon mgr.y on smithi067 2024-05-06T12:36:18.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:17 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3101951190' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/976842880"}]': finished 2024-05-06T12:36:18.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:17 smithi086 ceph-mon[23164]: osdmap e99: 8 total, 8 up, 8 in 2024-05-06T12:36:18.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:17 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2443342655' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]: dispatch 2024-05-06T12:36:18.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:17 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]: dispatch 2024-05-06T12:36:18.310 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:18 smithi067 systemd[1]: Stopping Ceph mgr.y for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:18.606 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:18 smithi067 podman[57596]: 2024-05-06 12:36:18.310214471 +0000 UTC m=+0.125157140 container died 156e546a6ad85f5765091e8f9e19132fce45286bbfe20f1dd89f4f1f5ff0c6d7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vendor=Red Hat, Inc.) 2024-05-06T12:36:18.949 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:18 smithi067 podman[57596]: 2024-05-06 12:36:18.70450114 +0000 UTC m=+0.519443815 container cleanup 156e546a6ad85f5765091e8f9e19132fce45286bbfe20f1dd89f4f1f5ff0c6d7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, version=8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, architecture=x86_64, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, release=754) 2024-05-06T12:36:18.949 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:18 smithi067 bash[57596]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y 2024-05-06T12:36:18.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[22413]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:18.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[22413]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]': finished 2024-05-06T12:36:18.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[22413]: osdmap e100: 8 total, 8 up, 8 in 2024-05-06T12:36:18.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1170308663' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3470008396"}]: dispatch 2024-05-06T12:36:18.950 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[27137]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:18.950 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[27137]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]': finished 2024-05-06T12:36:18.950 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[27137]: osdmap e100: 8 total, 8 up, 8 in 2024-05-06T12:36:18.950 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:18 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1170308663' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3470008396"}]: dispatch 2024-05-06T12:36:19.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:18 smithi086 ceph-mon[23164]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:19.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:18 smithi086 ceph-mon[23164]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1824486224"}]': finished 2024-05-06T12:36:19.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:18 smithi086 ceph-mon[23164]: osdmap e100: 8 total, 8 up, 8 in 2024-05-06T12:36:19.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:18 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1170308663' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3470008396"}]: dispatch 2024-05-06T12:36:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:18 smithi067 podman[57612]: 2024-05-06 12:36:18.947888189 +0000 UTC m=+0.640179289 container remove 156e546a6ad85f5765091e8f9e19132fce45286bbfe20f1dd89f4f1f5ff0c6d7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.expose-services=, maintainer=Guillaume Abrioux , release=754, vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0) 2024-05-06T12:36:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:18 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:36:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-05-06T12:36:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 systemd[1]: Stopped Ceph mgr.y for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service: Consumed 58.270s CPU time. 2024-05-06T12:36:19.533 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 systemd[1]: Starting Ceph mgr.y for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:19.533 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 podman[57769]: 2024-05-06 12:36:19.397940392 +0000 UTC m=+0.020143737 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:36:19.805 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 podman[57769]: 2024-05-06 12:36:19.532274903 +0000 UTC m=+0.154478231 container create c0f8319f1bf4db8b28d13e71d1898213ead589204408228f1abdad83a33c8fc8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-05-06T12:36:19.805 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 podman[57769]: 2024-05-06 12:36:19.776865549 +0000 UTC m=+0.399068877 container init c0f8319f1bf4db8b28d13e71d1898213ead589204408228f1abdad83a33c8fc8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-06T12:36:19.805 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 podman[57769]: 2024-05-06 12:36:19.779463394 +0000 UTC m=+0.401666722 container start c0f8319f1bf4db8b28d13e71d1898213ead589204408228f1abdad83a33c8fc8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, ceph=True, org.label-schema.schema-version=1.0) 2024-05-06T12:36:20.155 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 ceph-mgr[57807]: -- 172.21.15.67:0/3334338374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56405bf005a0 con 0x56405bed4c00 2024-05-06T12:36:20.156 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 bash[57769]: c0f8319f1bf4db8b28d13e71d1898213ead589204408228f1abdad83a33c8fc8 2024-05-06T12:36:20.156 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 systemd[1]: Started Ceph mgr.y for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:20.156 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:19 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:19.952+0000 7f03fe765200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:36:20.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:20.154+0000 7f03fe765200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:36:20.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:20.265+0000 7f03fe765200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:36:20.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/1170308663' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3470008396"}]': finished 2024-05-06T12:36:20.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[22413]: osdmap e101: 8 total, 8 up, 8 in 2024-05-06T12:36:20.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/171014436' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/289594580"}]: dispatch 2024-05-06T12:36:20.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[22413]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:20.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:20.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:20.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/1170308663' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3470008396"}]': finished 2024-05-06T12:36:20.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[27137]: osdmap e101: 8 total, 8 up, 8 in 2024-05-06T12:36:20.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/171014436' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/289594580"}]: dispatch 2024-05-06T12:36:20.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[27137]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:20.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:20.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:20 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:20.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:20 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/1170308663' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/3470008396"}]': finished 2024-05-06T12:36:20.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:20 smithi086 ceph-mon[23164]: osdmap e101: 8 total, 8 up, 8 in 2024-05-06T12:36:20.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:20 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/171014436' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/289594580"}]: dispatch 2024-05-06T12:36:20.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:20 smithi086 ceph-mon[23164]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:20.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:20 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:20.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:20 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:20.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:20.641+0000 7f03fe765200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:36:20.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:20 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:20.732+0000 7f03fe765200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:36:21.266 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ::ffff:172.21.15.86 - - [06/May/2024:12:36:20] "GET /metrics HTTP/1.1" 200 38139 "" "Prometheus/2.33.4" 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/171014436' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/289594580"}]': finished 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[22413]: osdmap e102: 8 total, 8 up, 8 in 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2183735627' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2116552457"}]: dispatch 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/171014436' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/289594580"}]': finished 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[27137]: osdmap e102: 8 total, 8 up, 8 in 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:21.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:21.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:21 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2183735627' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2116552457"}]: dispatch 2024-05-06T12:36:21.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:21 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/171014436' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/289594580"}]': finished 2024-05-06T12:36:21.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:21 smithi086 ceph-mon[23164]: osdmap e102: 8 total, 8 up, 8 in 2024-05-06T12:36:21.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:21 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:21.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:21 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:21.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:21 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2183735627' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2116552457"}]: dispatch 2024-05-06T12:36:22.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:22.049+0000 7f03fe765200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:36:22.604 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:22.249+0000 7f03fe765200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:36:22.604 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:22.439+0000 7f03fe765200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:36:22.604 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:22 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2183735627' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2116552457"}]': finished 2024-05-06T12:36:22.604 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:22 smithi067 ceph-mon[27137]: osdmap e103: 8 total, 8 up, 8 in 2024-05-06T12:36:22.604 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:22 smithi067 ceph-mon[27137]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:22.605 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:22 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2183735627' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2116552457"}]': finished 2024-05-06T12:36:22.605 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:22 smithi067 ceph-mon[22413]: osdmap e103: 8 total, 8 up, 8 in 2024-05-06T12:36:22.605 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:22 smithi067 ceph-mon[22413]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:22.855 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:22 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2183735627' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2116552457"}]': finished 2024-05-06T12:36:22.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:22 smithi086 ceph-mon[23164]: osdmap e103: 8 total, 8 up, 8 in 2024-05-06T12:36:22.856 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:22 smithi086 ceph-mon[23164]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:36:22.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:22.748+0000 7f03fe765200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:36:22.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:22.846+0000 7f03fe765200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:36:23.433 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:23.064+0000 7f03fe765200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:36:23.433 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:23.274+0000 7f03fe765200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:36:23.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:36:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:36:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:36:24.088 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:23.925+0000 7f03fe765200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:36:24.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:23 smithi067 ceph-mon[22413]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 1 op/s 2024-05-06T12:36:24.089 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:23 smithi067 ceph-mon[27137]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 1 op/s 2024-05-06T12:36:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:23 smithi086 ceph-mon[23164]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 935 B/s rd, 1 op/s 2024-05-06T12:36:24.713 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:24.382+0000 7f03fe765200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:36:24.713 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:24.489+0000 7f03fe765200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:36:24.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:24.723+0000 7f03fe765200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T12:36:25.644 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:25.341+0000 7f03fe765200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:36:25.644 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:25.444+0000 7f03fe765200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:36:25.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[22413]: from='client.25147 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:25.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[27137]: from='client.25147 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:25 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:25.715+0000 7f03fe765200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:25.862+0000 7f03fe765200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:36:25.969 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:25 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:25.967+0000 7f03fe765200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:36:26.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:25 smithi086 ceph-mon[23164]: from='client.25147 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:26.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:25 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:26.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:25 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:26.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:25 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:26.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:25 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:26.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:26 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:26.096+0000 7f03fe765200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:36:26.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:26 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:26.198+0000 7f03fe765200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:36:26.783 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:26 smithi067 ceph-mon[22413]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-05-06T12:36:26.783 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:26 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:26.783 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:26 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:26.784 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:26 smithi067 ceph-mon[27137]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-05-06T12:36:26.784 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:26 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:26.784 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:26 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:27.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:26 smithi086 ceph-mon[23164]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-05-06T12:36:27.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:26 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:27.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:26 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:27.421 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:27.080+0000 7f03fe765200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:36:27.421 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:27.311+0000 7f03fe765200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:36:27.422 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:27] ENGINE Bus STARTING 2024-05-06T12:36:27.422 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: CherryPy Checker: 2024-05-06T12:36:27.422 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: The Application mounted at '' has an empty config. 2024-05-06T12:36:27.422 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:36:27.422 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:27] ENGINE Serving on http://:::9283 2024-05-06T12:36:27.422 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:27] ENGINE Bus STARTED 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[22413]: Standby manager daemon y restarted 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[22413]: Standby manager daemon y started 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[27137]: Standby manager daemon y restarted 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[27137]: Standby manager daemon y started 2024-05-06T12:36:27.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T12:36:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:36:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T12:36:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:27 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:36:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:27 smithi086 ceph-mon[23164]: Standby manager daemon y restarted 2024-05-06T12:36:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:27 smithi086 ceph-mon[23164]: Standby manager daemon y started 2024-05-06T12:36:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:27 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T12:36:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:27 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:36:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:27 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T12:36:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:27 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:36:28.665 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:36:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:36:28.363Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=5 2024-05-06T12:36:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:28 smithi067 ceph-mon[22413]: mgrmap e28: x(active, since 60s), standbys: y 2024-05-06T12:36:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:28 smithi067 ceph-mon[22413]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T12:36:28.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:28 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:28.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:28 smithi067 ceph-mon[27137]: mgrmap e28: x(active, since 60s), standbys: y 2024-05-06T12:36:28.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:28 smithi067 ceph-mon[27137]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T12:36:28.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:28 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:29.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:28 smithi086 ceph-mon[23164]: mgrmap e28: x(active, since 60s), standbys: y 2024-05-06T12:36:29.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:28 smithi086 ceph-mon[23164]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T12:36:29.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:28 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:29.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:36:30.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: Reconfiguring iscsi.foo.smithi067.fnpkny (dependencies changed)... 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: Reconfiguring daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[22413]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:30.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:36:30.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: Reconfiguring iscsi.foo.smithi067.fnpkny (dependencies changed)... 2024-05-06T12:36:30.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:30.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:30.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:30.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: Reconfiguring daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:36:30.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:30 smithi067 ceph-mon[27137]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:36:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:36:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' 2024-05-06T12:36:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: Reconfiguring iscsi.foo.smithi067.fnpkny (dependencies changed)... 2024-05-06T12:36:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:36:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: Reconfiguring daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:36:30.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:30 smithi086 ceph-mon[23164]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:36:31.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:30 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ::ffff:172.21.15.86 - - [06/May/2024:12:36:30] "GET /metrics HTTP/1.1" 200 38136 "" "Prometheus/2.33.4" 2024-05-06T12:36:32.013 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:31 smithi067 ceph-mon[22413]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-05-06T12:36:32.013 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:31 smithi067 ceph-mon[27137]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-05-06T12:36:32.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:31 smithi086 ceph-mon[23164]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-05-06T12:36:33.224 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:33 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:33.224 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:33 smithi086 ceph-mon[23164]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T12:36:33.224 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:33 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T12:36:33.224 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:33 smithi086 ceph-mon[23164]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:32.957+0000 7fd3cb7b4700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Creating ceph-iscsi config... 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Traceback (most recent call last): 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return _run_code(code, main_globals, None, 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: exec(code, run_globals) 2024-05-06T12:36:33.225 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:33.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Traceback (most recent call last): 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return self.event_loop.get_result(coro, timeout) 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return future.result(timeout) 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return self.__get_result() 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: raise self._exception 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T12:36:33.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: use_current_daemon_image=reconfig, 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Creating ceph-iscsi config... 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Non-zero exit code 1 from systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: Traceback (most recent call last): 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: return _run_code(code, main_globals, None, 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:36:33.228 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: exec(code, run_globals) 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: RuntimeError: Failed command: systemctl restart ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:33.028+0000 7fd4391de700 -1 mgr handle_mgr_map I was active but no longer am 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ignoring --setuser ceph since I am not root 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ignoring --setgroup ceph since I am not root 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-mgr[44353]: -- 172.21.15.86:0/1030370015 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5574853045a0 con 0x5574852d8800 2024-05-06T12:36:33.229 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:33.222+0000 7f2c74528200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:36:33.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:33.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[22413]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T12:36:33.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T12:36:33.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[22413]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T12:36:33.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[27137]: from='mgr.24962 172.21.15.86:0/1344556217' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:33 smithi067 ceph-mon[27137]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:33] ENGINE Bus STOPPING 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:33] ENGINE Bus STOPPED 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:33] ENGINE Bus STARTING 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:33] ENGINE Serving on http://:::9283 2024-05-06T12:36:33.476 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:33] ENGINE Bus STARTED 2024-05-06T12:36:33.581 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:33.450+0000 7f2c74528200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:36:33.582 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:33.580+0000 7f2c74528200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:36:34.211 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:33.917+0000 7f2c74528200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:34 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:34.012+0000 7f2c74528200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.24962 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: mgrmap e29: y(active, starting, since 0.0556834s) 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:36:34.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: Manager daemon y is now available 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:36:34.213 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:34 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:36:34.322 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.24962 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: mgrmap e29: y(active, starting, since 0.0556834s) 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:36:34.323 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: Manager daemon y is now available 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.24962 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: mgrmap e29: y(active, starting, since 0.0556834s) 2024-05-06T12:36:34.324 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: Manager daemon y is now available 2024-05-06T12:36:34.325 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:34.326 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:36:34.326 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:34 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[27137]: mgrmap e30: y(active, since 1.06335s) 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[27137]: [06/May/2024:12:36:34] ENGINE Bus STARTING 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[27137]: [06/May/2024:12:36:34] ENGINE Serving on http://172.21.15.67:8765 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[27137]: [06/May/2024:12:36:34] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[27137]: [06/May/2024:12:36:34] ENGINE Bus STARTED 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[22413]: mgrmap e30: y(active, since 1.06335s) 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[22413]: [06/May/2024:12:36:34] ENGINE Bus STARTING 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[22413]: [06/May/2024:12:36:34] ENGINE Serving on http://172.21.15.67:8765 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[22413]: [06/May/2024:12:36:34] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:36:35.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:35 smithi067 ceph-mon[22413]: [06/May/2024:12:36:34] ENGINE Bus STARTED 2024-05-06T12:36:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:35 smithi086 ceph-mon[23164]: mgrmap e30: y(active, since 1.06335s) 2024-05-06T12:36:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:35 smithi086 ceph-mon[23164]: [06/May/2024:12:36:34] ENGINE Bus STARTING 2024-05-06T12:36:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:35 smithi086 ceph-mon[23164]: [06/May/2024:12:36:34] ENGINE Serving on http://172.21.15.67:8765 2024-05-06T12:36:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:35 smithi086 ceph-mon[23164]: [06/May/2024:12:36:34] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:36:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:35 smithi086 ceph-mon[23164]: [06/May/2024:12:36:34] ENGINE Bus STARTED 2024-05-06T12:36:35.615 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:35.242+0000 7f2c74528200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:36:35.616 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:35.425+0000 7f2c74528200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:36:35.616 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:35.614+0000 7f2c74528200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:36:36.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:36 smithi067 ceph-mon[22413]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:36.204 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:36 smithi067 ceph-mon[27137]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:36.217 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:36 smithi086 ceph-mon[23164]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:36.218 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:35.913+0000 7f2c74528200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:36:36.218 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:36.007+0000 7f2c74528200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:36:36.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:36.215+0000 7f2c74528200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:36:36.486 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:36.420+0000 7f2c74528200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:36:37.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:37 smithi086 ceph-mon[23164]: mgrmap e31: y(active, since 3s) 2024-05-06T12:36:37.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:37.051+0000 7f2c74528200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:36:37.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:37 smithi067 ceph-mon[22413]: mgrmap e31: y(active, since 3s) 2024-05-06T12:36:37.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:37 smithi067 ceph-mon[27137]: mgrmap e31: y(active, since 3s) 2024-05-06T12:36:37.811 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:37.490+0000 7f2c74528200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:36:37.811 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:37.594+0000 7f2c74528200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:36:37.811 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:37.810+0000 7f2c74528200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T12:36:38.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:38 smithi086 ceph-mon[23164]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:38.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:38 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:38 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:38 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:38 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:38.363+0000 7f2c74528200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:36:38.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:38.463+0000 7f2c74528200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:36:38.949 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:38.724+0000 7f2c74528200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:36:38.949 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:38.866+0000 7f2c74528200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:36:38.949 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[27137]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[22413]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:38 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:38.970+0000 7f2c74528200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:36:39.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:39.095+0000 7f2c74528200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:36:39.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:39.193+0000 7f2c74528200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:39.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:39 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:40.020 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:39 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:40.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:39 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:40.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:39 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:40.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:39 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:40.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:40.019+0000 7f2c74528200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:36:40.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:40.239+0000 7f2c74528200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:36:40.348 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:40] ENGINE Bus STARTING 2024-05-06T12:36:40.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: CherryPy Checker: 2024-05-06T12:36:40.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: The Application mounted at '' has an empty config. 2024-05-06T12:36:40.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: 2024-05-06T12:36:40.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:40] ENGINE Serving on http://:::9283 2024-05-06T12:36:40.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: [06/May/2024:12:36:40] ENGINE Bus STARTED 2024-05-06T12:36:40.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[22413]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:40.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[22413]: Standby manager daemon x started 2024-05-06T12:36:40.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:36:40.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:36:40.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:36:40.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:36:40.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[27137]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:40.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[27137]: Standby manager daemon x started 2024-05-06T12:36:40.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:36:40.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:36:40.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:36:40.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:40 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:36:41.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:40 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[44349]: ::ffff:172.21.15.86 - - [06/May/2024:12:36:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T12:36:41.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:40 smithi086 ceph-mon[23164]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:36:41.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:40 smithi086 ceph-mon[23164]: Standby manager daemon x started 2024-05-06T12:36:41.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:40 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:36:41.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:40 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:36:41.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:40 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:36:41.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:40 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/3549716544' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:36:42.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:42 smithi067 ceph-mon[22413]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-06T12:36:42.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:42 smithi067 ceph-mon[22413]: mgrmap e32: y(active, since 8s), standbys: x 2024-05-06T12:36:42.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:42 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:36:42.159 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:42 smithi067 ceph-mon[27137]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-06T12:36:42.159 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:42 smithi067 ceph-mon[27137]: mgrmap e32: y(active, since 8s), standbys: x 2024-05-06T12:36:42.159 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:42 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:36:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:42 smithi086 ceph-mon[23164]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-06T12:36:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:42 smithi086 ceph-mon[23164]: mgrmap e32: y(active, since 8s), standbys: x 2024-05-06T12:36:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:42 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:36:43.467 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:36:43.548 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.549 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.550 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:43 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:36:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:36:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:36:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:36:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:36:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:36:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:36:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:43 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.173 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:36:44.173 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (45s) 6s ago 5m 17.0M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (30s) 6s ago 5m 42.9M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 unknown 6s ago 4m - - 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283 running (85s) 6s ago 10m 68.9M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 68b03f3b238a 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (24s) 6s ago 12m 472M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (12m) 6s ago 12m 53.0M 2048M 17.2.0 e1d6a67b021e f5b02626cc9d 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (10m) 6s ago 10m 46.3M 2048M 17.2.0 e1d6a67b021e 5cb539dec1ee 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (10m) 6s ago 10m 40.3M 2048M 17.2.0 e1d6a67b021e d0561ab1fc3c 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (50s) 6s ago 6m 11.9M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (34s) 6s ago 6m 9457k - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (10m) 6s ago 10m 51.8M 2736M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (9m) 6s ago 9m 53.1M 2736M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (9m) 6s ago 9m 46.7M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (8m) 6s ago 8m 50.7M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:36:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (8m) 6s ago 8m 52.1M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (8m) 6s ago 8m 49.3M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (7m) 6s ago 7m 51.3M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (7m) 6s ago 7m 53.4M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (36s) 6s ago 6m 53.7M - 2.33.4 514e6a882f6e 92894a1d8e34 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (5m) 6s ago 5m 73.4M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (4m) 6s ago 4m 71.4M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (4m) 6s ago 4m 73.2M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:36:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (4m) 6s ago 4m 71.4M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:36:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:36:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-06T12:36:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:36:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[22413]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:36:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:44.477 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:44 smithi067 ceph-mon[27137]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:36:44.591 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:36:44.591 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-06T12:36:44.591 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:36:44.591 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.592 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.592 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-06T12:36:44.592 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:44.592 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T12:36:44.592 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:44.592 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:44 smithi086 ceph-mon[23164]: Reconfiguring daemon prometheus.a on smithi086 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": {}, 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:36:44.625 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:36:44.972 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:36:44.972 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:36:44.972 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "2/23 daemons upgraded", 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:36:44.973 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:36:45.384 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-06T12:36:45.384 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-06T12:36:45.384 INFO:teuthology.orchestra.run.smithi067.stdout: daemon iscsi.foo.smithi067.fnpkny on smithi067 is in unknown state 2024-05-06T12:36:45.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:45 smithi086 ceph-mon[23164]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:45.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:45 smithi086 ceph-mon[23164]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:45.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:45 smithi086 ceph-mon[23164]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:36:45.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:45 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2746344203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:36:45.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:45 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3607037296' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[22413]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[22413]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[22413]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/2746344203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/3607037296' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[27137]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[27137]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[27137]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2746344203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:36:45.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:45 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3607037296' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:36:46.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:46 smithi086 ceph-mon[23164]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:46.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:46 smithi086 ceph-mon[23164]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:46.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:46 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/4111511445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:36:46.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:46 smithi067 ceph-mon[22413]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:46.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:46 smithi067 ceph-mon[22413]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:46.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:46 smithi067 ceph-mon[22413]: from='client.? 172.21.15.67:0/4111511445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:36:46.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:46 smithi067 ceph-mon[27137]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:36:46.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:46 smithi067 ceph-mon[27137]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:46.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:46 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/4111511445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:36:47.044 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:46 smithi086 systemd[1]: Stopping Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:47.044 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:46 smithi086 bash[51000]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T12:36:47.336 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.042Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T12:36:47.336 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[47833]: ts=2024-05-06T12:36:47.043Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T12:36:47.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 podman[51013]: 2024-05-06 12:36:47.053124829 +0000 UTC m=+0.104334087 container died 92894a1d8e345e98d6b45512c32952a24b0faee7f43800c0b45d88263273c7e3 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:47.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 podman[51013]: 2024-05-06 12:36:47.433902075 +0000 UTC m=+0.485111331 container cleanup 92894a1d8e345e98d6b45512c32952a24b0faee7f43800c0b45d88263273c7e3 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:47.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 bash[51013]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T12:36:47.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 bash[51039]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T12:36:48.078 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 podman[51027]: 2024-05-06 12:36:47.727158769 +0000 UTC m=+0.675831007 container remove 92894a1d8e345e98d6b45512c32952a24b0faee7f43800c0b45d88263273c7e3 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:48.079 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T12:36:48.079 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 systemd[1]: Stopped Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:48.079 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 systemd[1]: Starting Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:48.079 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:47 smithi086 podman[51107]: 2024-05-06 12:36:47.87591741 +0000 UTC m=+0.019464135 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T12:36:48.079 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 podman[51107]: 2024-05-06 12:36:48.078228312 +0000 UTC m=+0.221775024 container create 8dffac3aef3bed9c386062779276327ec254daea6ffa73645080ba096e0ec185 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:48.364 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 podman[51107]: 2024-05-06 12:36:48.288727887 +0000 UTC m=+0.432274616 container init 8dffac3aef3bed9c386062779276327ec254daea6ffa73645080ba096e0ec185 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:48.364 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 podman[51107]: 2024-05-06 12:36:48.291383079 +0000 UTC m=+0.434929797 container start 8dffac3aef3bed9c386062779276327ec254daea6ffa73645080ba096e0ec185 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:36:48.364 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.313Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T12:36:48.364 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.313Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T12:36:48.364 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.313Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T12:36:48.364 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.313Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi086 (none))" 2024-05-06T12:36:48.364 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.313Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.313Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.315Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.316Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.317Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.319Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.319Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.863µs 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.319Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T12:36:48.365 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 bash[51107]: 8dffac3aef3bed9c386062779276327ec254daea6ffa73645080ba096e0ec185 2024-05-06T12:36:48.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 systemd[1]: Started Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:48.619 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.513Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-05-06T12:36:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:48 smithi086 ceph-mon[23164]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:48 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:48 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:48 smithi067 ceph-mon[22413]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:48 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:48 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:48 smithi067 ceph-mon[27137]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:48 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:48 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:36:48.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:48] ENGINE Bus STOPPING 2024-05-06T12:36:48.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:36:48.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:48] ENGINE Bus STOPPED 2024-05-06T12:36:48.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:48] ENGINE Bus STARTING 2024-05-06T12:36:48.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:48] ENGINE Serving on http://:::9283 2024-05-06T12:36:48.726 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:36:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:36:48] ENGINE Bus STARTED 2024-05-06T12:36:49.066 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.765Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.800Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.801Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.801Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.057µs wal_replay_duration=481.748302ms total_replay_duration=481.790476ms 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.805Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.805Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.805Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.823Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.453809ms db_storage=791ns remote_storage=1.667µs web_handler=315ns query_engine=781ns scrape=351.831µs scrape_sd=87.957µs notify=13.119µs notify_sd=13.815µs rules=16.722707ms 2024-05-06T12:36:49.067 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:36:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:36:48.823Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T12:36:49.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:49.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:49.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T12:36:49.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T12:36:49.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:49 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:49.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:49 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:36:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[22413]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:36:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:50.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[27137]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:50.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:50.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:36:50.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:50 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:50.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:50 smithi086 ceph-mon[23164]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:36:50.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:50 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:36:50.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:50 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:36:50.736 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:50 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:36:51.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:51 smithi086 ceph-mon[23164]: Upgrade: Updating mgr.x 2024-05-06T12:36:51.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:51 smithi086 ceph-mon[23164]: Deploying daemon mgr.x on smithi086 2024-05-06T12:36:51.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:51 smithi067 ceph-mon[22413]: Upgrade: Updating mgr.x 2024-05-06T12:36:51.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:51 smithi067 ceph-mon[22413]: Deploying daemon mgr.x on smithi086 2024-05-06T12:36:51.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:51 smithi067 ceph-mon[27137]: Upgrade: Updating mgr.x 2024-05-06T12:36:51.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:51 smithi067 ceph-mon[27137]: Deploying daemon mgr.x on smithi086 2024-05-06T12:36:52.176 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:51 smithi086 systemd[1]: Stopping Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:52.454 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 podman[51454]: 2024-05-06 12:36:52.174847799 +0000 UTC m=+0.138506293 container died 68b03f3b238a970af274dd2e3c83132490d2915b4ac489f3b213280933163990 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-70ce953, ceph=True, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-06T12:36:52.709 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:52 smithi086 ceph-mon[23164]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 498 B/s rd, 0 op/s 2024-05-06T12:36:52.709 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 podman[51454]: 2024-05-06 12:36:52.509327027 +0000 UTC m=+0.472985512 container cleanup 68b03f3b238a970af274dd2e3c83132490d2915b4ac489f3b213280933163990 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:36:52.709 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 bash[51454]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x 2024-05-06T12:36:52.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:52 smithi067 ceph-mon[22413]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 498 B/s rd, 0 op/s 2024-05-06T12:36:52.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:52 smithi067 ceph-mon[27137]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 498 B/s rd, 0 op/s 2024-05-06T12:36:53.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 podman[51468]: 2024-05-06 12:36:52.708735554 +0000 UTC m=+0.534750492 container remove 68b03f3b238a970af274dd2e3c83132490d2915b4ac489f3b213280933163990 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=squid-70ce953, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:36:53.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:36:53.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-06T12:36:53.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 systemd[1]: Stopped Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:53.120 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:52 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Consumed 27.422s CPU time. 2024-05-06T12:36:53.474 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 systemd[1]: Starting Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:36:53.474 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 podman[51598]: 2024-05-06 12:36:53.24453958 +0000 UTC m=+0.017932582 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:36:53.474 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 podman[51598]: 2024-05-06 12:36:53.368249909 +0000 UTC m=+0.141642932 container create cad655065a4e5282857ab2f04d6dd8654c12396a34a11185a890b09c19838d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:36:53.771 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 podman[51598]: 2024-05-06 12:36:53.696455265 +0000 UTC m=+0.469848242 container init cad655065a4e5282857ab2f04d6dd8654c12396a34a11185a890b09c19838d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:36:53.772 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 podman[51598]: 2024-05-06 12:36:53.699188723 +0000 UTC m=+0.472581701 container start cad655065a4e5282857ab2f04d6dd8654c12396a34a11185a890b09c19838d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:36:53.772 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 ceph-mgr[51617]: -- 172.21.15.86:0/1947610806 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590411be5a0 con 0x559041193000 2024-05-06T12:36:54.048 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 bash[51598]: cad655065a4e5282857ab2f04d6dd8654c12396a34a11185a890b09c19838d23 2024-05-06T12:36:54.049 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 systemd[1]: Started Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:36:54.049 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:53 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:53.854+0000 7f55a6e53200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:36:54.049 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:54 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:54.047+0000 7f55a6e53200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:36:54.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:54 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:54.155+0000 7f55a6e53200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:36:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:54 smithi067 ceph-mon[22413]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 498 B/s rd, 0 op/s 2024-05-06T12:36:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:54 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:54 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:54.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:54 smithi067 ceph-mon[27137]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 498 B/s rd, 0 op/s 2024-05-06T12:36:54.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:54 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:54.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:54 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:54.797 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:54 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:54.504+0000 7f55a6e53200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:36:54.798 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:54 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:54.588+0000 7f55a6e53200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:36:54.798 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:54 smithi086 ceph-mon[23164]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 498 B/s rd, 0 op/s 2024-05-06T12:36:54.798 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:54 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:54.798 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:54 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:55.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:55 smithi086 ceph-mon[23164]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:55.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:55 smithi067 ceph-mon[22413]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:55.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:55 smithi067 ceph-mon[27137]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:36:56.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:55 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:55.835+0000 7f55a6e53200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:36:56.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:56 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:56.027+0000 7f55a6e53200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:36:56.201 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:56 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:56.199+0000 7f55a6e53200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:36:56.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:56 smithi067 ceph-mon[22413]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:56.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:56 smithi067 ceph-mon[27137]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:56.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:56 smithi086 ceph-mon[23164]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:56.823 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:56 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:56.506+0000 7f55a6e53200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:36:56.824 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:56 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:56.603+0000 7f55a6e53200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:36:56.824 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:56 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:56.822+0000 7f55a6e53200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:36:57.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:57 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:57.023+0000 7f55a6e53200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:36:57.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:57 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:57.647+0000 7f55a6e53200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:36:58.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:58.085+0000 7f55a6e53200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:36:58.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:58.192+0000 7f55a6e53200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:36:58.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:58 smithi067 ceph-mon[22413]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:58.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:58 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:58.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:36:58 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:58.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:58 smithi067 ceph-mon[27137]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:58.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:58 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:58.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:36:58 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:58.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:58.407+0000 7f55a6e53200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T12:36:58.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:58 smithi086 ceph-mon[23164]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:36:58.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:58 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:58.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:36:58 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:36:59.226 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:58.954+0000 7f55a6e53200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:36:59.227 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:59 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:59.052+0000 7f55a6e53200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:36:59.564 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:59 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:59.319+0000 7f55a6e53200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:36:59.564 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:59 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:59.461+0000 7f55a6e53200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:36:59.565 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:59 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:59.563+0000 7f55a6e53200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:36:59.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:59 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:59.686+0000 7f55a6e53200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:36:59.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:36:59 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:36:59.791+0000 7f55a6e53200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:37:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:00 smithi086 ceph-mon[23164]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:00 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:00 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:00 smithi067 ceph-mon[22413]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:00 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:00 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:00.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:00 smithi067 ceph-mon[27137]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:00.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:00 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:00.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:00 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:00.628+0000 7f55a6e53200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:00.852+0000 7f55a6e53200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: [06/May/2024:12:37:00] ENGINE Bus STARTING 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: CherryPy Checker: 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: The Application mounted at '' has an empty config. 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: [06/May/2024:12:37:00] ENGINE Serving on http://:::9283 2024-05-06T12:37:00.961 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:00 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: [06/May/2024:12:37:00] ENGINE Bus STARTED 2024-05-06T12:37:01.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:01 smithi086 ceph-mon[23164]: Standby manager daemon x restarted 2024-05-06T12:37:01.674 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:01 smithi086 ceph-mon[23164]: Standby manager daemon x started 2024-05-06T12:37:01.674 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:01 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:37:01.675 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:01 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:37:01.675 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:01 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:37:01.675 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:01 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[22413]: Standby manager daemon x restarted 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[22413]: Standby manager daemon x started 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[22413]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[27137]: Standby manager daemon x restarted 2024-05-06T12:37:01.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[27137]: Standby manager daemon x started 2024-05-06T12:37:01.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:37:01.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:37:01.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:37:01.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:01 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/47470416' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:37:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:02 smithi086 ceph-mon[23164]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:02 smithi086 ceph-mon[23164]: mgrmap e33: y(active, since 28s), standbys: x 2024-05-06T12:37:02.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:02 smithi067 ceph-mon[22413]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:02.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:02 smithi067 ceph-mon[22413]: mgrmap e33: y(active, since 28s), standbys: x 2024-05-06T12:37:02.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:02 smithi067 ceph-mon[27137]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:02.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:02 smithi067 ceph-mon[27137]: mgrmap e33: y(active, since 28s), standbys: x 2024-05-06T12:37:03.873 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:37:03] "GET /metrics HTTP/1.1" 200 38227 "" "Prometheus/2.33.4" 2024-05-06T12:37:03.873 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.873 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: Upgrade: Setting container_image for all mgr 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-06T12:37:03.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: Upgrade: It appears safe to stop mon.a 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:03.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: Upgrade: Setting container_image for all mgr 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: Upgrade: It appears safe to stop mon.a 2024-05-06T12:37:03.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:03 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: Upgrade: Setting container_image for all mgr 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-06T12:37:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-06T12:37:04.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:04.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-06T12:37:04.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: Upgrade: It appears safe to stop mon.a 2024-05-06T12:37:04.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:03 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:05.790 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[27137]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:05.790 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[27137]: Upgrade: Updating mon.a 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[27137]: Deploying daemon mon.a on smithi067 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[22413]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[22413]: Upgrade: Updating mon.a 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[22413]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:05.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:05 smithi067 ceph-mon[22413]: Deploying daemon mon.a on smithi067 2024-05-06T12:37:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:05 smithi086 ceph-mon[23164]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:05 smithi086 ceph-mon[23164]: Upgrade: Updating mon.a 2024-05-06T12:37:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:05 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' 2024-05-06T12:37:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:05 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:05 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:05 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:05 smithi086 ceph-mon[23164]: Deploying daemon mon.a on smithi067 2024-05-06T12:37:06.757 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:06 smithi067 systemd[1]: Stopping Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:37:06.757 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:06 smithi067 ceph-mon[22413]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:06.757 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:06 smithi067 ceph-mon[27137]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:07.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:06 smithi086 ceph-mon[23164]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:07.132 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:06] ENGINE Bus STOPPING 2024-05-06T12:37:07.132 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:37:07.132 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:06] ENGINE Bus STOPPED 2024-05-06T12:37:07.132 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:06] ENGINE Bus STARTING 2024-05-06T12:37:07.132 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:06] ENGINE Serving on http://:::9283 2024-05-06T12:37:07.132 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:06] ENGINE Bus STARTED 2024-05-06T12:37:07.132 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a[22409]: 2024-05-06T12:37:06.754+0000 7f08dacd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:37:07.133 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:06 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a[22409]: 2024-05-06T12:37:06.754+0000 7f08dacd4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-06T12:37:07.133 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:06 smithi067 podman[61891]: 2024-05-06 12:37:06.816471668 +0000 UTC m=+0.166010284 container died f5b02626cc9d13f6a751addaf5eeb80b78c29865215e5d9e4a9cad98b5889c17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, ceph=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vcs-type=git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, release=754, name=centos-stream) 2024-05-06T12:37:07.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:07 smithi067 podman[61891]: 2024-05-06 12:37:07.222294737 +0000 UTC m=+0.571833350 container cleanup f5b02626cc9d13f6a751addaf5eeb80b78c29865215e5d9e4a9cad98b5889c17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, ceph=True, distribution-scope=public, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64) 2024-05-06T12:37:07.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:07 smithi067 bash[61891]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a 2024-05-06T12:37:07.843 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:07 smithi067 podman[61916]: 2024-05-06 12:37:07.48209092 +0000 UTC m=+0.661731843 container remove f5b02626cc9d13f6a751addaf5eeb80b78c29865215e5d9e4a9cad98b5889c17 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, io.openshift.expose-services=, version=8, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, RELEASE=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.buildah.version=1.19.8) 2024-05-06T12:37:07.843 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:07 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-06T12:37:07.844 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:07 smithi067 systemd[1]: Stopped Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:37:07.844 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:07 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service: Consumed 15.579s CPU time. 2024-05-06T12:37:08.198 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:07 smithi067 systemd[1]: Starting Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:37:08.198 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 podman[62051]: 2024-05-06 12:37:07.965817501 +0000 UTC m=+0.019468846 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:37:08.198 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 podman[62051]: 2024-05-06 12:37:08.07503767 +0000 UTC m=+0.128689044 container create 60a77a61933a66d5e9d7fcb15c3346f03dd3c96444222cab174b614e87525c7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-06T12:37:08.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 podman[62051]: 2024-05-06 12:37:08.319671653 +0000 UTC m=+0.373323006 container init 60a77a61933a66d5e9d7fcb15c3346f03dd3c96444222cab174b614e87525c7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-06T12:37:08.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 podman[62051]: 2024-05-06 12:37:08.323190629 +0000 UTC m=+0.376841967 container start 60a77a61933a66d5e9d7fcb15c3346f03dd3c96444222cab174b614e87525c7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-06T12:37:08.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T12:37:08.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev), process ceph-mon, pid 2 2024-05-06T12:37:08.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: pidfile_write: ignore empty --pid-file 2024-05-06T12:37:08.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: load: jerasure load: lrc 2024-05-06T12:37:08.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: RocksDB version: 7.9.2 2024-05-06T12:37:08.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Git sha 0 2024-05-06T12:37:08.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Compile date 2024-05-03 15:08:47 2024-05-06T12:37:08.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: DB SUMMARY 2024-05-06T12:37:08.477 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: DB Session ID: OIXBEYV50DIEKRUG6WTJ 2024-05-06T12:37:08.478 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: CURRENT file: CURRENT 2024-05-06T12:37:08.478 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T12:37:08.478 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: MANIFEST file: MANIFEST-000015 size: 972 Bytes 2024-05-06T12:37:08.478 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-05-06T12:37:08.478 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3430215 ; 2024-05-06T12:37:08.478 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.error_if_exists: 0 2024-05-06T12:37:08.478 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.create_if_missing: 0 2024-05-06T12:37:08.479 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.paranoid_checks: 1 2024-05-06T12:37:08.479 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-06T12:37:08.479 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T12:37:08.479 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-06T12:37:08.479 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.env: 0x55f58bd9ec60 2024-05-06T12:37:08.479 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.fs: PosixFileSystem 2024-05-06T12:37:08.479 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.info_log: 0x55f58de91ac0 2024-05-06T12:37:08.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T12:37:08.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.statistics: (nil) 2024-05-06T12:37:08.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.use_fsync: 0 2024-05-06T12:37:08.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_log_file_size: 0 2024-05-06T12:37:08.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T12:37:08.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T12:37:08.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T12:37:08.481 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T12:37:08.481 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.allow_fallocate: 1 2024-05-06T12:37:08.481 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T12:37:08.481 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T12:37:08.481 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.use_direct_reads: 0 2024-05-06T12:37:08.481 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T12:37:08.482 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T12:37:08.482 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.db_log_dir: 2024-05-06T12:37:08.482 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.wal_dir: 2024-05-06T12:37:08.482 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T12:37:08.482 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T12:37:08.482 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T12:37:08.482 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T12:37:08.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T12:37:08.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T12:37:08.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T12:37:08.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T12:37:08.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.write_buffer_manager: 0x55f58de641e0 2024-05-06T12:37:08.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T12:37:08.484 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T12:37:08.484 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T12:37:08.484 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T12:37:08.484 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T12:37:08.484 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T12:37:08.484 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T12:37:08.484 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T12:37:08.485 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.unordered_write: 0 2024-05-06T12:37:08.485 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T12:37:08.485 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T12:37:08.485 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T12:37:08.485 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T12:37:08.485 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.row_cache: None 2024-05-06T12:37:08.486 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.wal_filter: None 2024-05-06T12:37:08.486 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T12:37:08.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T12:37:08.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.two_write_queues: 0 2024-05-06T12:37:08.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T12:37:08.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.wal_compression: 0 2024-05-06T12:37:08.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.atomic_flush: 0 2024-05-06T12:37:08.489 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T12:37:08.489 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T12:37:08.489 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T12:37:08.489 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.log_readahead_size: 0 2024-05-06T12:37:08.489 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T12:37:08.489 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T12:37:08.489 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T12:37:08.490 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T12:37:08.490 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T12:37:08.490 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T12:37:08.490 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-06T12:37:08.490 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_background_jobs: 2 2024-05-06T12:37:08.490 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_background_compactions: -1 2024-05-06T12:37:08.491 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_subcompactions: 1 2024-05-06T12:37:08.491 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T12:37:08.491 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T12:37:08.491 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T12:37:08.491 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T12:37:08.491 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T12:37:08.492 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T12:37:08.492 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T12:37:08.492 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T12:37:08.492 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_open_files: -1 2024-05-06T12:37:08.492 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T12:37:08.492 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T12:37:08.492 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T12:37:08.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T12:37:08.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_background_flushes: -1 2024-05-06T12:37:08.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Compression algorithms supported: 2024-05-06T12:37:08.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T12:37:08.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kZSTD supported: 0 2024-05-06T12:37:08.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kXpressCompression supported: 0 2024-05-06T12:37:08.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T12:37:08.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kZlibCompression supported: 1 2024-05-06T12:37:08.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kSnappyCompression supported: 1 2024-05-06T12:37:08.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kLZ4Compression supported: 1 2024-05-06T12:37:08.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: kBZip2Compression supported: 0 2024-05-06T12:37:08.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T12:37:08.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-06T12:37:08.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-05-06T12:37:08.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-06T12:37:08.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T12:37:08.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.merge_operator: 2024-05-06T12:37:08.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_filter: None 2024-05-06T12:37:08.496 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T12:37:08.496 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T12:37:08.496 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T12:37:08.496 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T12:37:08.496 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f58de91bc0) 2024-05-06T12:37:08.496 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-05-06T12:37:08.496 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T12:37:08.497 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T12:37:08.497 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-05-06T12:37:08.497 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_type: 0 2024-05-06T12:37:08.497 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_index_type: 0 2024-05-06T12:37:08.497 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_shortening: 1 2024-05-06T12:37:08.497 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T12:37:08.498 INFO:journalctl@ceph.mon.a.smithi067.stdout: checksum: 4 2024-05-06T12:37:08.498 INFO:journalctl@ceph.mon.a.smithi067.stdout: no_block_cache: 0 2024-05-06T12:37:08.498 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache: 0x55f58de95090 2024-05-06T12:37:08.498 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-05-06T12:37:08.498 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_options: 2024-05-06T12:37:08.499 INFO:journalctl@ceph.mon.a.smithi067.stdout: capacity : 536870912 2024-05-06T12:37:08.499 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_shard_bits : 4 2024-05-06T12:37:08.499 INFO:journalctl@ceph.mon.a.smithi067.stdout: strict_capacity_limit : 0 2024-05-06T12:37:08.499 INFO:journalctl@ceph.mon.a.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-05-06T12:37:08.499 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_compressed: (nil) 2024-05-06T12:37:08.499 INFO:journalctl@ceph.mon.a.smithi067.stdout: persistent_cache: (nil) 2024-05-06T12:37:08.499 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size: 4096 2024-05-06T12:37:08.500 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size_deviation: 10 2024-05-06T12:37:08.500 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_restart_interval: 16 2024-05-06T12:37:08.500 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_block_restart_interval: 1 2024-05-06T12:37:08.500 INFO:journalctl@ceph.mon.a.smithi067.stdout: metadata_block_size: 4096 2024-05-06T12:37:08.500 INFO:journalctl@ceph.mon.a.smithi067.stdout: partition_filters: 0 2024-05-06T12:37:08.500 INFO:journalctl@ceph.mon.a.smithi067.stdout: use_delta_encoding: 1 2024-05-06T12:37:08.501 INFO:journalctl@ceph.mon.a.smithi067.stdout: filter_policy: bloomfilter 2024-05-06T12:37:08.501 INFO:journalctl@ceph.mon.a.smithi067.stdout: whole_key_filtering: 1 2024-05-06T12:37:08.501 INFO:journalctl@ceph.mon.a.smithi067.stdout: verify_compression: 0 2024-05-06T12:37:08.501 INFO:journalctl@ceph.mon.a.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-05-06T12:37:08.501 INFO:journalctl@ceph.mon.a.smithi067.stdout: format_version: 5 2024-05-06T12:37:08.501 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_index_compression: 1 2024-05-06T12:37:08.501 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_align: 0 2024-05-06T12:37:08.502 INFO:journalctl@ceph.mon.a.smithi067.stdout: max_auto_readahead_size: 262144 2024-05-06T12:37:08.502 INFO:journalctl@ceph.mon.a.smithi067.stdout: prepopulate_block_cache: 0 2024-05-06T12:37:08.502 INFO:journalctl@ceph.mon.a.smithi067.stdout: initial_auto_readahead_size: 8192 2024-05-06T12:37:08.502 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-05-06T12:37:08.502 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T12:37:08.502 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression: NoCompression 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.num_levels: 7 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T12:37:08.503 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-06T12:37:08.504 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T12:37:08.505 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.arena_block_size: 1048576 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T12:37:08.506 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T12:37:08.507 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T12:37:08.507 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T12:37:08.507 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T12:37:08.507 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.table_properties_collectors: 2024-05-06T12:37:08.509 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.inplace_update_support: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.bloom_locality: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.max_successive_merges: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.ttl: 2592000 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-06T12:37:08.510 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.enable_blob_files: false 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.min_blob_size: 0 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.blob_file_starting_level: 0 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12074, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-05-06T12:37:08.511 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a1b807f9-3aff-4cdc-8971-efb529c25cbf 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999028353556, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999028363135, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3149705, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12075, "largest_seqno": 13577, "table_properties": {"data_size": 3142452, "index_size": 4570, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 15891, "raw_average_key_size": 23, "raw_value_size": 3127855, "raw_average_value_size": 4710, "num_data_blocks": 207, "num_entries": 664, "num_filter_entries": 664, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714999028, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a1b807f9-3aff-4cdc-8971-efb529c25cbf", "db_session_id": "OIXBEYV50DIEKRUG6WTJ", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999028363240, "job": 1, "event": "recovery_finished"} 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f58df86000 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: DB pointer 0x55f58df70000 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** DB Stats ** 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:08.512 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:08.513 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:37:08.513 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T12:37:08.513 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:08.513 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:37:08.513 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: L0 1/0 3.00 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 666.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: L6 1/0 7.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: Sum 2/0 10.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 666.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 666.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 666.3 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:37:08.514 INFO:journalctl@ceph.mon.a.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative compaction: 0.00 GB write, 212.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval compaction: 0.00 GB write, 212.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache BinnedLRUCache@0x55f58de95090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.7e-05 secs_since: 0 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T12:37:08.515 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: starting mon.a rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???) e3 preinit fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???).mds e1 new map 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???).mds e1 print_map 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout: e1 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout: legacy client fscid: -1 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout: No filesystems configured 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:08.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 ceph-mon[62070]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-05-06T12:37:08.517 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 bash[62051]: 60a77a61933a66d5e9d7fcb15c3346f03dd3c96444222cab174b614e87525c7b 2024-05-06T12:37:08.517 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:08 smithi067 systemd[1]: Started Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:37:08.922 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:37:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:37:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:37:11.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:11 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ignoring --setuser ceph since I am not root 2024-05-06T12:37:11.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:11 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ignoring --setgroup ceph since I am not root 2024-05-06T12:37:11.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mgr[57807]: -- 172.21.15.67:0/1740263126 <== mon.1 v2:172.21.15.67:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648197485a0 con 0x56481971cc00 2024-05-06T12:37:11.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:11 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:11.594+0000 7f33b4d2b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 537 B/s rd, 0 op/s 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: mon.a calling monitor election 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: monmap epoch 3 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: last_changed 2024-05-06T12:26:06.449836+0000 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: min_mon_release 17 (quincy) 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: election_strategy: 1 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: fsmap 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: mgrmap e33: y(active, since 38s), standbys: x 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: daemon iscsi.foo.smithi067.fnpkny on smithi067 is in unknown state 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: from='mgr.25198 ' entity='' 2024-05-06T12:37:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[62070]: mgrmap e34: y(active, since 38s), standbys: x 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 537 B/s rd, 0 op/s 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: mon.a calling monitor election 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: monmap epoch 3 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: last_changed 2024-05-06T12:26:06.449836+0000 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: min_mon_release 17 (quincy) 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: election_strategy: 1 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:11.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: fsmap 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: mgrmap e33: y(active, since 38s), standbys: x 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: daemon iscsi.foo.smithi067.fnpkny on smithi067 is in unknown state 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: from='mgr.25198 ' entity='' 2024-05-06T12:37:11.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:11 smithi067 ceph-mon[27137]: mgrmap e34: y(active, since 38s), standbys: x 2024-05-06T12:37:11.754 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: ignoring --setuser ceph since I am not root 2024-05-06T12:37:11.754 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: ignoring --setgroup ceph since I am not root 2024-05-06T12:37:11.754 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mgr[51617]: -- 172.21.15.86:0/2025475817 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56436380e5a0 con 0x5643637e3000 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:11.569+0000 7fbdb338d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:11.752+0000 7fbdb338d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: from='mgr.25198 172.21.15.67:0/1324080290' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 537 B/s rd, 0 op/s 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: mon.a calling monitor election 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: monmap epoch 3 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: last_changed 2024-05-06T12:26:06.449836+0000 2024-05-06T12:37:11.755 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: min_mon_release 17 (quincy) 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: election_strategy: 1 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: fsmap 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: mgrmap e33: y(active, since 38s), standbys: x 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: daemon iscsi.foo.smithi067.fnpkny on smithi067 is in unknown state 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: from='mgr.25198 ' entity='' 2024-05-06T12:37:11.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:11 smithi086 ceph-mon[23164]: mgrmap e34: y(active, since 38s), standbys: x 2024-05-06T12:37:12.119 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:11.853+0000 7fbdb338d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:37:12.216 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:11 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:11.786+0000 7f33b4d2b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:37:12.216 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:11 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:11.892+0000 7f33b4d2b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T12:37:12.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:12 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:12.215+0000 7f33b4d2b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:37:12.475 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:12 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:12.303+0000 7f33b4d2b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:37:12.622 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:12.158+0000 7fbdb338d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T12:37:12.622 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:12.242+0000 7fbdb338d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T12:37:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:13 smithi067 ceph-mon[62070]: mgrmap e35: y(active, since 39s), standbys: x 2024-05-06T12:37:13.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:13.563+0000 7f33b4d2b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:37:13.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:13 smithi067 ceph-mon[27137]: mgrmap e35: y(active, since 39s), standbys: x 2024-05-06T12:37:13.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:13 smithi086 ceph-mon[23164]: mgrmap e35: y(active, since 39s), standbys: x 2024-05-06T12:37:13.763 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:13.419+0000 7fbdb338d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T12:37:13.763 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:13.590+0000 7fbdb338d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:37:14.053 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:13.761+0000 7fbdb338d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:37:14.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:13.742+0000 7f33b4d2b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T12:37:14.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:13.922+0000 7f33b4d2b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T12:37:14.352 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:14.051+0000 7fbdb338d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:37:14.352 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:14.146+0000 7fbdb338d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:37:14.540 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:14 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:14.226+0000 7f33b4d2b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T12:37:14.541 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:14 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:14.324+0000 7f33b4d2b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T12:37:14.541 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:14 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:14.539+0000 7f33b4d2b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:37:14.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:14.350+0000 7fbdb338d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T12:37:14.619 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:14.546+0000 7fbdb338d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:37:14.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:14 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:14.741+0000 7f33b4d2b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T12:37:15.568 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:15.141+0000 7fbdb338d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:37:15.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:15 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:15.375+0000 7f33b4d2b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T12:37:15.869 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:15.566+0000 7fbdb338d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:37:15.870 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:15.664+0000 7fbdb338d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:37:16.151 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:15 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:15.834+0000 7f33b4d2b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T12:37:16.152 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:15 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:15.936+0000 7f33b4d2b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T12:37:16.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:15.870+0000 7fbdb338d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T12:37:16.474 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:16 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:16.150+0000 7f33b4d2b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T12:37:16.745 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:16.395+0000 7fbdb338d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:37:16.745 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:16.492+0000 7fbdb338d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:37:16.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:16 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:16.704+0000 7f33b4d2b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T12:37:16.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:16 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:16.802+0000 7f33b4d2b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T12:37:17.104 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:16.743+0000 7fbdb338d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:37:17.105 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:16.883+0000 7fbdb338d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:37:17.105 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:16.983+0000 7fbdb338d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:37:17.105 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:17.103+0000 7fbdb338d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:37:17.369 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:17.201+0000 7fbdb338d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:37:17.430 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:17.061+0000 7f33b4d2b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T12:37:17.430 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:17.203+0000 7f33b4d2b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T12:37:17.430 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:17.305+0000 7f33b4d2b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T12:37:17.430 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:17.428+0000 7f33b4d2b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T12:37:17.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:17.528+0000 7f33b4d2b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T12:37:18.311 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:18.020+0000 7fbdb338d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:37:18.311 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:18.242+0000 7fbdb338d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:37:18.311 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: [06/May/2024:12:37:18] ENGINE Bus STARTING 2024-05-06T12:37:18.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:18 smithi086 ceph-mon[23164]: Standby manager daemon x restarted 2024-05-06T12:37:18.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:18 smithi086 ceph-mon[23164]: Standby manager daemon x started 2024-05-06T12:37:18.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:18 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:37:18.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:18 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:37:18.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:18 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:37:18.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:18 smithi086 ceph-mon[23164]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:37:18.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: CherryPy Checker: 2024-05-06T12:37:18.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: The Application mounted at '' has an empty config. 2024-05-06T12:37:18.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: 2024-05-06T12:37:18.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: [06/May/2024:12:37:18] ENGINE Serving on http://:::9283 2024-05-06T12:37:18.620 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:37:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x[51613]: [06/May/2024:12:37:18] ENGINE Bus STARTED 2024-05-06T12:37:18.623 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:18.390+0000 7f33b4d2b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T12:37:18.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[27137]: Standby manager daemon x restarted 2024-05-06T12:37:18.623 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[27137]: Standby manager daemon x started 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[27137]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[62070]: Standby manager daemon x restarted 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[62070]: Standby manager daemon x started 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[62070]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[62070]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[62070]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T12:37:18.624 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:18 smithi067 ceph-mon[62070]: from='mgr.? 172.21.15.86:0/2127887042' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T12:37:18.874 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:18.621+0000 7f33b4d2b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T12:37:18.875 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:18] ENGINE Bus STARTING 2024-05-06T12:37:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: CherryPy Checker: 2024-05-06T12:37:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: The Application mounted at '' has an empty config. 2024-05-06T12:37:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:18] ENGINE Serving on http://:::9283 2024-05-06T12:37:19.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:18 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:18] ENGINE Bus STARTED 2024-05-06T12:37:19.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: mgrmap e36: y(active, since 45s), standbys: x 2024-05-06T12:37:19.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: Active manager daemon y restarted 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: Activating manager daemon y 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: mgrmap e37: y(active, starting, since 0.0101037s), standbys: x 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:37:19.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: Manager daemon y is now available 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:37:19.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:19 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:37:19.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: mgrmap e36: y(active, since 45s), standbys: x 2024-05-06T12:37:19.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: Active manager daemon y restarted 2024-05-06T12:37:19.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: Activating manager daemon y 2024-05-06T12:37:19.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:19.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: mgrmap e37: y(active, starting, since 0.0101037s), standbys: x 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:37:19.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: Manager daemon y is now available 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: mgrmap e36: y(active, since 45s), standbys: x 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: Active manager daemon y restarted 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: Activating manager daemon y 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: mgrmap e37: y(active, starting, since 0.0101037s), standbys: x 2024-05-06T12:37:19.653 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T12:37:19.654 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T12:37:19.655 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T12:37:19.655 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: Manager daemon y is now available 2024-05-06T12:37:19.655 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:19.655 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:37:19.655 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T12:37:19.655 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:37:19.655 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:19 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T12:37:19.673 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:37:19.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:19 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:19.649+0000 7f337218f700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-06T12:37:20.361 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:37:20.361 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (81s) 42s ago 6m 17.0M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:37:20.361 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (66s) 22s ago 6m 43.0M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:37:20.361 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 unknown 42s ago 5m - - 2024-05-06T12:37:20.361 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (27s) 22s ago 10m 64.1M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:37:20.361 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (60s) 42s ago 12m 472M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (12m) 42s ago 12m 53.0M 2048M 17.2.0 e1d6a67b021e f5b02626cc9d 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (11m) 22s ago 11m 46.7M 2048M 17.2.0 e1d6a67b021e 5cb539dec1ee 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (11m) 42s ago 11m 40.3M 2048M 17.2.0 e1d6a67b021e d0561ab1fc3c 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (87s) 42s ago 7m 11.9M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (71s) 22s ago 7m 13.8M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (10m) 42s ago 10m 51.8M 2736M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (10m) 42s ago 10m 53.1M 2736M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (9m) 42s ago 9m 46.7M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (9m) 42s ago 9m 50.7M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (9m) 22s ago 9m 52.5M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (8m) 22s ago 8m 49.5M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (8m) 22s ago 8m 51.7M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (7m) 22s ago 7m 53.9M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (32s) 22s ago 7m 42.6M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:37:20.362 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (5m) 42s ago 5m 73.4M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:37:20.363 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (5m) 22s ago 5m 71.9M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:37:20.363 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (5m) 42s ago 5m 73.2M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:37:20.363 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (5m) 22s ago 5m 71.8M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:37:20.805 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": {}, 2024-05-06T12:37:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:37:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:37:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:37:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-05-06T12:37:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:37:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:37:20.807 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:37:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: [06/May/2024:12:37:19] ENGINE Bus STARTING 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: mgrmap e38: y(active, since 1.01431s), standbys: x 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: [06/May/2024:12:37:19] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: [06/May/2024:12:37:19] ENGINE Serving on http://172.21.15.67:8765 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: [06/May/2024:12:37:19] ENGINE Bus STARTED 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:20 smithi086 ceph-mon[23164]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: [06/May/2024:12:37:19] ENGINE Bus STARTING 2024-05-06T12:37:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: mgrmap e38: y(active, since 1.01431s), standbys: x 2024-05-06T12:37:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: [06/May/2024:12:37:19] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:37:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: [06/May/2024:12:37:19] ENGINE Serving on http://172.21.15.67:8765 2024-05-06T12:37:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: [06/May/2024:12:37:19] ENGINE Bus STARTED 2024-05-06T12:37:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[27137]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: [06/May/2024:12:37:19] ENGINE Bus STARTING 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: mgrmap e38: y(active, since 1.01431s), standbys: x 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: [06/May/2024:12:37:19] ENGINE Serving on https://172.21.15.67:7150 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: [06/May/2024:12:37:19] ENGINE Serving on http://172.21.15.67:8765 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: [06/May/2024:12:37:19] ENGINE Bus STARTED 2024-05-06T12:37:20.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:20.978 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:20 smithi067 ceph-mon[62070]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:37:21.160 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:37:21.160 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "2/23 daemons upgraded", 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:37:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:37:21.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:21 smithi086 ceph-mon[23164]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-06T12:37:21.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:21 smithi086 ceph-mon[23164]: Cluster is now healthy 2024-05-06T12:37:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:21 smithi086 ceph-mon[23164]: mgrmap e39: y(active, since 2s), standbys: x 2024-05-06T12:37:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:21 smithi086 ceph-mon[23164]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:21 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/2803705469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:21 smithi086 ceph-mon[23164]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:21.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:21 smithi086 ceph-mon[23164]: from='client.? 172.21.15.67:0/3404467448' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:37:21.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[27137]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-06T12:37:21.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[27137]: Cluster is now healthy 2024-05-06T12:37:21.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[27137]: mgrmap e39: y(active, since 2s), standbys: x 2024-05-06T12:37:21.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[27137]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/2803705469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[27137]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[27137]: from='client.? 172.21.15.67:0/3404467448' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[62070]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[62070]: mgrmap e39: y(active, since 2s), standbys: x 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[62070]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/2803705469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[62070]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:21.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:21 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3404467448' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:37:23.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:37:23] "GET /metrics HTTP/1.1" 200 35442 "" "Prometheus/2.33.4" 2024-05-06T12:37:23.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[27137]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:23.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[27137]: mgrmap e40: y(active, since 4s), standbys: x 2024-05-06T12:37:23.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:23.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:23.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[62070]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:23.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[62070]: mgrmap e40: y(active, since 4s), standbys: x 2024-05-06T12:37:23.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:23.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:23 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:23 smithi086 ceph-mon[23164]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:37:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:23 smithi086 ceph-mon[23164]: mgrmap e40: y(active, since 4s), standbys: x 2024-05-06T12:37:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:23 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:23 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:24 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:24 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:24 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:24 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:24.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:24 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:26.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:25 smithi067 ceph-mon[62070]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-05-06T12:37:26.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:25 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:26.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:25 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:26.073 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:25 smithi067 ceph-mon[27137]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-05-06T12:37:26.073 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:25 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:26.073 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:25 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:26.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:25 smithi086 ceph-mon[23164]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-05-06T12:37:26.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:25 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:26.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:25 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:27 smithi086 ceph-mon[23164]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T12:37:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:27 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:27 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:27 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:28.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:27 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:28.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[62070]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T12:37:28.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:28.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:28.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:28.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:28.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[27137]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T12:37:28.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:28.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:28.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:28.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:27 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:29.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:37:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:37:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:37:29.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:37:29.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:28 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:37:29.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:37:29.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:37:29.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:37:29.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:37:29.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-06T12:37:29.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:29.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:28 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:37:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:37:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T12:37:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:30.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:30.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:30.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-06T12:37:30.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:29 smithi086 ceph-mon[23164]: Upgrade: It appears safe to stop mon.c 2024-05-06T12:37:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:37:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:37:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T12:37:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[62070]: Upgrade: It appears safe to stop mon.c 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: Updating smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: Updating smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-06T12:37:30.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:29 smithi067 ceph-mon[27137]: Upgrade: It appears safe to stop mon.c 2024-05-06T12:37:30.830 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[27137]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:30.831 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[27137]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:30.831 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:30.831 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:30 smithi086 ceph-mon[23164]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:30 smithi086 ceph-mon[23164]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:30 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:30 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:31.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:30 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:31.164 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[27137]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:31.164 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:31.165 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[62070]: from='mgr.25309 ' entity='mgr.y' 2024-05-06T12:37:31.165 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:31.165 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:31.165 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:32.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:31 smithi086 ceph-mon[23164]: Upgrade: Updating mon.c 2024-05-06T12:37:32.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:31 smithi086 ceph-mon[23164]: Deploying daemon mon.c on smithi067 2024-05-06T12:37:32.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:31 smithi086 ceph-mon[23164]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:37:32.150 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:31 smithi067 ceph-mon[27137]: Upgrade: Updating mon.c 2024-05-06T12:37:32.150 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:31 smithi067 ceph-mon[27137]: Deploying daemon mon.c on smithi067 2024-05-06T12:37:32.150 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:31 smithi067 ceph-mon[27137]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:37:32.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:31 smithi067 ceph-mon[62070]: Upgrade: Updating mon.c 2024-05-06T12:37:32.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:31 smithi067 ceph-mon[62070]: Deploying daemon mon.c on smithi067 2024-05-06T12:37:32.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:31 smithi067 ceph-mon[62070]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T12:37:32.706 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:32 smithi067 systemd[1]: Stopping Ceph mon.c for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:37:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:32] ENGINE Bus STOPPING 2024-05-06T12:37:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:37:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:32] ENGINE Bus STOPPED 2024-05-06T12:37:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:32] ENGINE Bus STARTING 2024-05-06T12:37:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:32] ENGINE Serving on http://:::9283 2024-05-06T12:37:32.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:37:32] ENGINE Bus STARTED 2024-05-06T12:37:32.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c[27133]: 2024-05-06T12:37:32.703+0000 7f3439468700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:37:32.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c[27133]: 2024-05-06T12:37:32.703+0000 7f3439468700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-06T12:37:32.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:32 smithi067 podman[64455]: 2024-05-06 12:37:32.836586691 +0000 UTC m=+0.244410058 container died d0561ab1fc3ce617bcc6b9ac11f061c8dcf3e52720a51ef078259a2b4c20fa0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, name=centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8) 2024-05-06T12:37:33.389 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:33 smithi067 podman[64455]: 2024-05-06 12:37:33.203578519 +0000 UTC m=+0.611401873 container cleanup d0561ab1fc3ce617bcc6b9ac11f061c8dcf3e52720a51ef078259a2b4c20fa0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_CLEAN=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, name=centos-stream, RELEASE=HEAD) 2024-05-06T12:37:33.389 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:33 smithi067 bash[64455]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c 2024-05-06T12:37:33.643 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:33 smithi067 podman[64481]: 2024-05-06 12:37:33.388198033 +0000 UTC m=+0.552525978 container remove d0561ab1fc3ce617bcc6b9ac11f061c8dcf3e52720a51ef078259a2b4c20fa0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., name=centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, version=8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, release=754) 2024-05-06T12:37:33.643 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:33 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.c.service: Deactivated successfully. 2024-05-06T12:37:33.643 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:33 smithi067 systemd[1]: Stopped Ceph mon.c for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:37:33.643 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:33 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.c.service: Consumed 8.097s CPU time. 2024-05-06T12:37:33.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:37:33] "GET /metrics HTTP/1.1" 200 35442 "" "Prometheus/2.33.4" 2024-05-06T12:37:33.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:33 smithi067 systemd[1]: Starting Ceph mon.c for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:37:34.326 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 podman[64615]: 2024-05-06 12:37:33.95660891 +0000 UTC m=+0.018512395 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:37:34.326 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 podman[64615]: 2024-05-06 12:37:34.089902574 +0000 UTC m=+0.151806049 container create 120694fd2a6c84d883187c0a36c195f7495f9f062ac2ac47a0b791ebfb448437 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-05-06T12:37:34.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 podman[64615]: 2024-05-06 12:37:34.325769997 +0000 UTC m=+0.387673474 container init 120694fd2a6c84d883187c0a36c195f7495f9f062ac2ac47a0b791ebfb448437 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-06T12:37:34.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 podman[64615]: 2024-05-06 12:37:34.329246244 +0000 UTC m=+0.391149720 container start 120694fd2a6c84d883187c0a36c195f7495f9f062ac2ac47a0b791ebfb448437 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev), process ceph-mon, pid 2 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: pidfile_write: ignore empty --pid-file 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: load: jerasure load: lrc 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: RocksDB version: 7.9.2 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Git sha 0 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Compile date 2024-05-03 15:08:47 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: DB SUMMARY 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: DB Session ID: HL4AQT6RVXFRB5ZDZQQ0 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: CURRENT file: CURRENT 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-05-06T12:37:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 752369 ; 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.error_if_exists: 0 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.create_if_missing: 0 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.paranoid_checks: 1 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.env: 0x55f5e37cec60 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.fs: PosixFileSystem 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.info_log: 0x55f5e4b97ac0 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T12:37:34.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.statistics: (nil) 2024-05-06T12:37:34.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.use_fsync: 0 2024-05-06T12:37:34.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_log_file_size: 0 2024-05-06T12:37:34.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.allow_fallocate: 1 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.use_direct_reads: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.db_log_dir: 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.wal_dir: 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T12:37:34.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.write_buffer_manager: 0x55f5e4b6a1e0 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T12:37:34.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.unordered_write: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.row_cache: None 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.wal_filter: None 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.two_write_queues: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.wal_compression: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.atomic_flush: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T12:37:34.731 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.log_readahead_size: 0 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_background_jobs: 2 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_background_compactions: -1 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_subcompactions: 1 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T12:37:34.732 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_open_files: -1 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_background_flushes: -1 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Compression algorithms supported: 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kZSTD supported: 0 2024-05-06T12:37:34.733 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kXpressCompression supported: 0 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kZlibCompression supported: 1 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kSnappyCompression supported: 1 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kLZ4Compression supported: 1 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: kBZip2Compression supported: 0 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.merge_operator: 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_filter: None 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T12:37:34.734 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f5e4b97bc0) 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: index_type: 0 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: data_block_index_type: 0 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: index_shortening: 1 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: checksum: 4 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: no_block_cache: 0 2024-05-06T12:37:34.735 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache: 0x55f5e4b9b090 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache_options: 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: capacity : 536870912 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: num_shard_bits : 4 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: strict_capacity_limit : 0 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_cache_compressed: (nil) 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: persistent_cache: (nil) 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_size: 4096 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_size_deviation: 10 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_restart_interval: 16 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: index_block_restart_interval: 1 2024-05-06T12:37:34.736 INFO:journalctl@ceph.mon.c.smithi067.stdout: metadata_block_size: 4096 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: partition_filters: 0 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: use_delta_encoding: 1 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: filter_policy: bloomfilter 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: whole_key_filtering: 1 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: verify_compression: 0 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: format_version: 5 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: enable_index_compression: 1 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: block_align: 0 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: max_auto_readahead_size: 262144 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: prepopulate_block_cache: 0 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: initial_auto_readahead_size: 8192 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T12:37:34.737 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression: NoCompression 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.num_levels: 7 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T12:37:34.738 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T12:37:34.739 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.arena_block_size: 1048576 2024-05-06T12:37:34.740 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.table_properties_collectors: 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.inplace_update_support: 0 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T12:37:34.741 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.bloom_locality: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.max_successive_merges: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.ttl: 2592000 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.enable_blob_files: false 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.min_blob_size: 0 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T12:37:34.742 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.blob_file_starting_level: 0 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14891, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0fce8468-99a5-480b-a826-ca7541cfc140 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999054358991, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999054361601, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 443837, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14896, "largest_seqno": 15147, "table_properties": {"data_size": 441648, "index_size": 857, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2765, "raw_average_key_size": 24, "raw_value_size": 438928, "raw_average_value_size": 3850, "num_data_blocks": 37, "num_entries": 114, "num_filter_entries": 114, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714999054, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0fce8468-99a5-480b-a826-ca7541cfc140", "db_session_id": "HL4AQT6RVXFRB5ZDZQQ0", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999054361690, "job": 1, "event": "recovery_finished"} 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-05-06T12:37:34.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T12:37:34.744 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T12:37:34.744 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f5e4c8c000 2024-05-06T12:37:34.744 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: DB pointer 0x55f5e4c76000 2024-05-06T12:37:34.744 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-06T12:37:34.744 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-06T12:37:34.744 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** DB Stats ** 2024-05-06T12:37:34.744 INFO:journalctl@ceph.mon.c.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: L0 1/0 433.43 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 299.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: L6 1/0 11.21 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Sum 2/0 11.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 299.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 299.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T12:37:34.745 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 299.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: Cumulative compaction: 0.00 GB write, 88.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:37:34.746 INFO:journalctl@ceph.mon.c.smithi067.stdout: Interval compaction: 0.00 GB write, 88.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: Block cache BinnedLRUCache@0x55f5e4b9b090#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.95 KB,0.000181794%) Misc(1,0.00 KB,0%) 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: starting mon.c rank 1 at public addrs [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] at bind addrs [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???) e3 preinit fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???).mds e1 new map 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???).mds e1 print_map 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: e1 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: legacy client fscid: -1 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout: No filesystems configured 2024-05-06T12:37:34.747 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-06T12:37:34.748 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:34.748 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:34.748 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:34.748 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 ceph-mon[64635]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-05-06T12:37:34.748 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 bash[64615]: 120694fd2a6c84d883187c0a36c195f7495f9f062ac2ac47a0b791ebfb448437 2024-05-06T12:37:34.748 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:34 smithi067 systemd[1]: Started Ceph mon.c for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:37:36.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: mon.c calling monitor election 2024-05-06T12:37:36.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:37:36.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: mon.a calling monitor election 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: monmap epoch 3 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: last_changed 2024-05-06T12:26:06.449836+0000 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: min_mon_release 17 (quincy) 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: election_strategy: 1 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: fsmap 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:36.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: mgrmap e40: y(active, since 17s), standbys: x 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: overall HEALTH_OK 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: mon.c calling monitor election 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: mon.a calling monitor election 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: monmap epoch 3 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: last_changed 2024-05-06T12:26:06.449836+0000 2024-05-06T12:37:36.967 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: min_mon_release 17 (quincy) 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: election_strategy: 1 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: fsmap 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: mgrmap e40: y(active, since 17s), standbys: x 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: overall HEALTH_OK 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:36.968 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:36 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:37.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: mon.c calling monitor election 2024-05-06T12:37:37.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T12:37:37.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: mon.a calling monitor election 2024-05-06T12:37:37.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: monmap epoch 3 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: last_changed 2024-05-06T12:26:06.449836+0000 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: min_mon_release 17 (quincy) 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: election_strategy: 1 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: fsmap 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: mgrmap e40: y(active, since 17s), standbys: x 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: overall HEALTH_OK 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:37.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:37.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:36 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:38.022 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:37 smithi086 ceph-mon[23164]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:38.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:37 smithi067 ceph-mon[64635]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:38.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:37 smithi067 ceph-mon[62070]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:39.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:37:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:37:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:37:39.664 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:39 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:39.369+0000 7f337218f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-06T12:37:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:40 smithi067 ceph-mon[62070]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:40.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:40 smithi067 ceph-mon[64635]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:40.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:40 smithi086 ceph-mon[23164]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:41 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:41 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:41 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:41.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:41 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:41.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:41 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:41.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:41 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:41.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:41 smithi086 ceph-mon[23164]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:37:41.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:41 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:41.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:41 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:42 smithi067 ceph-mon[62070]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:42 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:42 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:42 smithi067 ceph-mon[64635]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:42 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:42.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:42 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:42.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:42 smithi086 ceph-mon[23164]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:42.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:42 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:42.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:42 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:44.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:37:43] "GET /metrics HTTP/1.1" 200 38175 "" "Prometheus/2.33.4" 2024-05-06T12:37:44.508 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:44 smithi067 ceph-mon[62070]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:44.508 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:44 smithi067 ceph-mon[64635]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:44.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:44 smithi086 ceph-mon[23164]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:46.169 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:46.169 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.169 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-06T12:37:46.170 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:45 smithi086 ceph-mon[23164]: Upgrade: It appears safe to stop mon.b 2024-05-06T12:37:46.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:46.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[64635]: Upgrade: It appears safe to stop mon.b 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:46.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-06T12:37:46.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:45 smithi067 ceph-mon[62070]: Upgrade: It appears safe to stop mon.b 2024-05-06T12:37:48.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:47 smithi086 ceph-mon[23164]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:48.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:47 smithi086 ceph-mon[23164]: Upgrade: Updating mon.b 2024-05-06T12:37:48.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:47 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:48.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:47 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:47 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:47 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:48.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:47 smithi086 ceph-mon[23164]: Deploying daemon mon.b on smithi086 2024-05-06T12:37:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[62070]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[62070]: Upgrade: Updating mon.b 2024-05-06T12:37:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:48.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[62070]: Deploying daemon mon.b on smithi086 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[64635]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[64635]: Upgrade: Updating mon.b 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:37:48.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:47 smithi067 ceph-mon[64635]: Deploying daemon mon.b on smithi086 2024-05-06T12:37:48.742 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:37:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:37:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:37:48.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:48 smithi086 systemd[1]: Stopping Ceph mon.b for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:37:48.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:48 smithi086 ceph-mon[23164]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:49.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:49.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:37:49.353 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b[23159]: 2024-05-06T12:37:48.991+0000 7fdd53125700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:37:49.354 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b[23159]: 2024-05-06T12:37:48.991+0000 7fdd53125700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-06T12:37:49.354 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:49 smithi086 podman[55088]: 2024-05-06 12:37:49.053507491 +0000 UTC m=+0.169629074 container died 5cb539dec1ee6309d1477e53df297bcbb9fe241b31d3b917b169a65c4f8514de (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, release=754, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-05-06T12:37:49.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:49 smithi086 podman[55088]: 2024-05-06 12:37:49.459415021 +0000 UTC m=+0.575536591 container cleanup 5cb539dec1ee6309d1477e53df297bcbb9fe241b31d3b917b169a65c4f8514de (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vcs-type=git, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vendor=Red Hat, Inc., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-06T12:37:49.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:49 smithi086 bash[55088]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b 2024-05-06T12:37:50.044 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:49 smithi086 podman[55102]: 2024-05-06 12:37:49.710840478 +0000 UTC m=+0.655992998 container remove 5cb539dec1ee6309d1477e53df297bcbb9fe241b31d3b917b169a65c4f8514de (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.openshift.expose-services=, version=8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, release=754, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870) 2024-05-06T12:37:50.045 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:49 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-06T12:37:50.045 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:49 smithi086 systemd[1]: Stopped Ceph mon.b for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:37:50.045 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:49 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.b.service: Consumed 9.147s CPU time. 2024-05-06T12:37:50.045 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 systemd[1]: Starting Ceph mon.b for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:37:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 podman[55233]: 2024-05-06 12:37:50.162557366 +0000 UTC m=+0.018177473 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:37:50.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 podman[55233]: 2024-05-06 12:37:50.287281693 +0000 UTC m=+0.142901775 container create 3215d78b6d70e2fff1839bfccb375dd3a511698dbd0f049c611601c44a40fff7 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953) 2024-05-06T12:37:50.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 podman[55233]: 2024-05-06 12:37:50.531864618 +0000 UTC m=+0.387484710 container init 3215d78b6d70e2fff1839bfccb375dd3a511698dbd0f049c611601c44a40fff7 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:37:50.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 podman[55233]: 2024-05-06 12:37:50.535187908 +0000 UTC m=+0.390807996 container start 3215d78b6d70e2fff1839bfccb375dd3a511698dbd0f049c611601c44a40fff7 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953) 2024-05-06T12:37:50.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T12:37:50.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev), process ceph-mon, pid 2 2024-05-06T12:37:50.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: pidfile_write: ignore empty --pid-file 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: load: jerasure load: lrc 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: RocksDB version: 7.9.2 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Git sha 0 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Compile date 2024-05-03 15:08:47 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: DB SUMMARY 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: DB Session ID: R8VLSSI16B9L4A31U7SV 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: CURRENT file: CURRENT 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1486697 ; 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.error_if_exists: 0 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.create_if_missing: 0 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.paranoid_checks: 1 2024-05-06T12:37:50.805 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-06T12:37:50.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T12:37:50.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-06T12:37:50.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.env: 0x5621148f9c60 2024-05-06T12:37:50.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.fs: PosixFileSystem 2024-05-06T12:37:50.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.info_log: 0x562116153ac0 2024-05-06T12:37:50.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T12:37:50.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.statistics: (nil) 2024-05-06T12:37:50.807 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.use_fsync: 0 2024-05-06T12:37:50.807 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_log_file_size: 0 2024-05-06T12:37:50.807 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T12:37:50.807 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.allow_fallocate: 1 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.use_direct_reads: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.db_log_dir: 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.wal_dir: 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T12:37:50.808 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.write_buffer_manager: 0x5621161261e0 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.unordered_write: 0 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T12:37:50.809 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.row_cache: None 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.wal_filter: None 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.two_write_queues: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.wal_compression: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.atomic_flush: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T12:37:50.810 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.log_readahead_size: 0 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_background_jobs: 2 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_background_compactions: -1 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_subcompactions: 1 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T12:37:50.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_open_files: -1 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_background_flushes: -1 2024-05-06T12:37:50.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Compression algorithms supported: 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kZSTD supported: 0 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kXpressCompression supported: 0 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kZlibCompression supported: 1 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kSnappyCompression supported: 1 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kLZ4Compression supported: 1 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: kBZip2Compression supported: 0 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.merge_operator: 2024-05-06T12:37:50.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_filter: None 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562116153bc0) 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_type: 0 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: data_block_index_type: 0 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_shortening: 1 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T12:37:50.814 INFO:journalctl@ceph.mon.b.smithi086.stdout: checksum: 4 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: no_block_cache: 0 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache: 0x562116157090 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_options: 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: capacity : 536870912 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: num_shard_bits : 4 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: strict_capacity_limit : 0 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_compressed: (nil) 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: persistent_cache: (nil) 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_size: 4096 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_size_deviation: 10 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_restart_interval: 16 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_block_restart_interval: 1 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: metadata_block_size: 4096 2024-05-06T12:37:50.815 INFO:journalctl@ceph.mon.b.smithi086.stdout: partition_filters: 0 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: use_delta_encoding: 1 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: filter_policy: bloomfilter 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: whole_key_filtering: 1 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: verify_compression: 0 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: format_version: 5 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: enable_index_compression: 1 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_align: 0 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: max_auto_readahead_size: 262144 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: prepopulate_block_cache: 0 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: initial_auto_readahead_size: 8192 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T12:37:50.816 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression: NoCompression 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.num_levels: 7 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-06T12:37:50.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T12:37:50.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-06T12:37:50.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.arena_block_size: 1048576 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.table_properties_collectors: 2024-05-06T12:37:50.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.inplace_update_support: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.bloom_locality: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.max_successive_merges: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.ttl: 2592000 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-06T12:37:50.821 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.enable_blob_files: false 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.min_blob_size: 0 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.blob_file_starting_level: 0 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14810, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-05-06T12:37:50.822 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 210d0ecd-c5be-44e3-8a71-0a939686f7f0 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999070564944, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999070570849, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 863699, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14815, "largest_seqno": 15354, "table_properties": {"data_size": 860432, "index_size": 1613, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 5497, "raw_average_key_size": 23, "raw_value_size": 855231, "raw_average_value_size": 3702, "num_data_blocks": 73, "num_entries": 231, "num_filter_entries": 231, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714999070, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "210d0ecd-c5be-44e3-8a71-0a939686f7f0", "db_session_id": "R8VLSSI16B9L4A31U7SV", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714999070570925, "job": 1, "event": "recovery_finished"} 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562116248000 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: DB pointer 0x562116232000 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** DB Stats ** 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:50.823 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** Compaction Stats [default] ** 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: L0 1/0 843.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 295.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: L6 1/0 11.21 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: Sum 2/0 12.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 295.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 295.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-06T12:37:50.824 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** Compaction Stats [default] ** 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 295.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative compaction: 0.00 GB write, 78.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval compaction: 0.00 GB write, 78.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T12:37:50.825 INFO:journalctl@ceph.mon.b.smithi086.stdout: Block cache BinnedLRUCache@0x562116157090#2 capacity: 512.00 MB usage: 2.55 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.72 KB,0.000137091%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: starting mon.b rank 2 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???) e3 preinit fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???).mds e1 new map 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???).mds e1 print_map 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: e1 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: legacy client fscid: -1 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout: No filesystems configured 2024-05-06T12:37:50.826 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-06T12:37:50.827 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:50.827 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:50.827 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T12:37:50.827 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-05-06T12:37:50.827 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 bash[55233]: 3215d78b6d70e2fff1839bfccb375dd3a511698dbd0f049c611601c44a40fff7 2024-05-06T12:37:50.827 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 systemd[1]: Started Ceph mon.b for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:37:50.827 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:50 smithi086 ceph-mon[55252]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:51.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:50 smithi067 ceph-mon[62070]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:51.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:50 smithi067 ceph-mon[64635]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:54.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:37:53] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.33.4" 2024-05-06T12:37:55.890 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:37:55 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:37:55.581+0000 7f337218f700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-06T12:37:56.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: fsmap 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mgrmap e40: y(active, since 32s), standbys: x 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: overall HEALTH_OK 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mon.c calling monitor election 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mon.a calling monitor election 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mon.b calling monitor election 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:56.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: overall HEALTH_OK 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mon.a calling monitor election 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mon.c calling monitor election 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: monmap epoch 4 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: last_changed 2024-05-06T12:37:50.804500+0000 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: min_mon_release 19 (squid) 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: election_strategy: 1 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:56.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: fsmap 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: mgrmap e40: y(active, since 37s), standbys: x 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: overall HEALTH_OK 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: fsmap 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mgrmap e40: y(active, since 32s), standbys: x 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: overall HEALTH_OK 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mon.c calling monitor election 2024-05-06T12:37:56.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mon.a calling monitor election 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mon.b calling monitor election 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: overall HEALTH_OK 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mon.a calling monitor election 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mon.c calling monitor election 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: monmap epoch 4 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: last_changed 2024-05-06T12:37:50.804500+0000 2024-05-06T12:37:56.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: min_mon_release 19 (squid) 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: election_strategy: 1 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: fsmap 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: mgrmap e40: y(active, since 37s), standbys: x 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: overall HEALTH_OK 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:56.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:56.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: fsmap 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mgrmap e40: y(active, since 32s), standbys: x 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: overall HEALTH_OK 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mon.c calling monitor election 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mon.a calling monitor election 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mon.b calling monitor election 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:56.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: overall HEALTH_OK 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mon.a calling monitor election 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mon.c calling monitor election 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: monmap epoch 4 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: fsid 847b865c-0ba3-11ef-bc97-c7b262605968 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: last_changed 2024-05-06T12:37:50.804500+0000 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: created 2024-05-06T12:24:31.741125+0000 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: min_mon_release 19 (squid) 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: election_strategy: 1 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: 1: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-06T12:37:56.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: fsmap 2024-05-06T12:37:56.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T12:37:56.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: mgrmap e40: y(active, since 37s), standbys: x 2024-05-06T12:37:56.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: overall HEALTH_OK 2024-05-06T12:37:56.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:56.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:37:56.382 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:37:57.065 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:37:57.065 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (118s) 16s ago 6m 17.1M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:37:57.065 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (103s) 33s ago 6m 43.6M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:37:57.065 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (73s) 16s ago 5m 49.8M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:37:57.065 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (63s) 33s ago 11m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:37:57.065 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (97s) 16s ago 13m 493M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:37:57.065 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (49s) 16s ago 13m 39.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 starting - - - 2048M 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (23s) 16s ago 11m 18.4M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (2m) 16s ago 7m 16.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (107s) 33s ago 7m 16.7M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (11m) 16s ago 11m 52.6M 2736M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (10m) 16s ago 10m 53.8M 2736M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (10m) 16s ago 10m 47.7M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (10m) 16s ago 10m 51.5M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (9m) 33s ago 9m 52.8M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (9m) 33s ago 9m 49.7M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (8m) 33s ago 8m 51.9M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (8m) 33s ago 8m 54.0M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (69s) 33s ago 7m 55.4M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (6m) 16s ago 6m 74.7M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:37:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (6m) 33s ago 6m 72.0M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:37:57.067 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (6m) 16s ago 6m 74.2M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:37:57.067 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (6m) 33s ago 6m 72.1M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:37:57.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:56 smithi086 ceph-mon[55252]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:57.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:56 smithi067 ceph-mon[62070]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:57.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:56 smithi067 ceph-mon[64635]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:57.479 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:37:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:37:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:37:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:37:57.480 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:37:57.826 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:37:57.826 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "4/23 daemons upgraded", 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mon daemons", 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:37:57.827 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:37:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[62070]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[62070]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[62070]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/813934482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:58.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[64635]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:58.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[64635]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[64635]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:57 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/813934482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:58.241 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:37:58.298 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:57 smithi086 ceph-mon[55252]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:58.298 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:57 smithi086 ceph-mon[55252]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:58.299 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:57 smithi086 ceph-mon[55252]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:58.299 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:57 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/813934482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:37:58.913 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:37:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:37:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:37:59.219 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:58 smithi086 ceph-mon[55252]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:59.219 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:58 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/4123462255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:37:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:58 smithi067 ceph-mon[62070]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:58 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/4123462255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:37:59.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:58 smithi067 ceph-mon[64635]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:37:59.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:58 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/4123462255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:37:59.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:37:59 smithi067 ceph-mon[62070]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:37:59.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:37:59 smithi067 ceph-mon[64635]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:37:59 smithi086 ceph-mon[55252]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:01.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:01 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:01.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:01 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:01.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:01 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:01.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:01 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:01.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:01 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:01.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:01 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:01.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:01 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:01.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:01 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:01.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:01 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:02 smithi086 ceph-mon[55252]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:02 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:02.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:02 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:02.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:02 smithi067 ceph-mon[64635]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:02.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:02 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:02.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:02 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:02.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:02 smithi067 ceph-mon[62070]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:02.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:02 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:02.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:02 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:04.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:38:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:38:03] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.33.4" 2024-05-06T12:38:04.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:04 smithi086 ceph-mon[55252]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:04.615 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:04.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:04 smithi067 ceph-mon[62070]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:04.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:04 smithi067 ceph-mon[64635]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:05.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: Reconfiguring mon.a (monmap changed)... 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[62070]: Reconfiguring daemon mon.a on smithi067 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:05.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:05.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:38:05.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:05.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: Reconfiguring mon.a (monmap changed)... 2024-05-06T12:38:05.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:05.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:05.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:05.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:05 smithi067 ceph-mon[64635]: Reconfiguring daemon mon.a on smithi067 2024-05-06T12:38:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:38:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: Reconfiguring mon.a (monmap changed)... 2024-05-06T12:38:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:05 smithi086 ceph-mon[55252]: Reconfiguring daemon mon.a on smithi067 2024-05-06T12:38:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:38:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:38:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:07.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:07.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:07.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:38:07.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:38:07.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:38:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:38:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: Reconfiguring mgr.y (monmap changed)... 2024-05-06T12:38:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: Reconfiguring daemon mgr.y on smithi067 2024-05-06T12:38:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:08.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:08.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:08.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:08.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:08.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: Reconfiguring mgr.y (monmap changed)... 2024-05-06T12:38:08.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: Reconfiguring daemon mgr.y on smithi067 2024-05-06T12:38:08.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:08.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:08.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:08.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:08.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:08.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:08.771 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:38:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:38:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:38:08.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: Reconfiguring mgr.y (monmap changed)... 2024-05-06T12:38:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: Reconfiguring daemon mgr.y on smithi067 2024-05-06T12:38:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:08.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:08.773 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:08.773 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:09.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:09 smithi067 ceph-mon[62070]: Reconfiguring mon.c (monmap changed)... 2024-05-06T12:38:09.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:09 smithi067 ceph-mon[62070]: Reconfiguring daemon mon.c on smithi067 2024-05-06T12:38:09.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:09 smithi067 ceph-mon[64635]: Reconfiguring mon.c (monmap changed)... 2024-05-06T12:38:09.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:09 smithi067 ceph-mon[64635]: Reconfiguring daemon mon.c on smithi067 2024-05-06T12:38:09.782 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:09 smithi086 ceph-mon[55252]: Reconfiguring mon.c (monmap changed)... 2024-05-06T12:38:09.782 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:09 smithi086 ceph-mon[55252]: Reconfiguring daemon mon.c on smithi067 2024-05-06T12:38:10.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[64635]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:10.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[62070]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:38:10.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:10 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:10 smithi086 ceph-mon[55252]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:10 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:10.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:10 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:10.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:10 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:38:10.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:10 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:11.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:11.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[64635]: Reconfiguring osd.0 (monmap changed)... 2024-05-06T12:38:11.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.0 on smithi067 2024-05-06T12:38:11.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:11.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:11.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:38:11.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:11.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[62070]: Reconfiguring osd.0 (monmap changed)... 2024-05-06T12:38:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.0 on smithi067 2024-05-06T12:38:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:38:11.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:11 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:11 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:11 smithi086 ceph-mon[55252]: Reconfiguring osd.0 (monmap changed)... 2024-05-06T12:38:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:11 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.0 on smithi067 2024-05-06T12:38:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:11 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:11.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:11 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:11.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:11 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:38:11.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:11 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:12.700 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:12 smithi067 ceph-mon[64635]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:12.701 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:12 smithi067 ceph-mon[64635]: Reconfiguring osd.1 (monmap changed)... 2024-05-06T12:38:12.701 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:12 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.1 on smithi067 2024-05-06T12:38:12.701 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:12 smithi067 ceph-mon[62070]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:12.701 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:12 smithi067 ceph-mon[62070]: Reconfiguring osd.1 (monmap changed)... 2024-05-06T12:38:12.701 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:12 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.1 on smithi067 2024-05-06T12:38:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:12 smithi086 ceph-mon[55252]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:12 smithi086 ceph-mon[55252]: Reconfiguring osd.1 (monmap changed)... 2024-05-06T12:38:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:12 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.1 on smithi067 2024-05-06T12:38:13.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:38:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:38:13] "GET /metrics HTTP/1.1" 200 38254 "" "Prometheus/2.33.4" 2024-05-06T12:38:14.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:14 smithi086 ceph-mon[55252]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:14.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:14 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:14.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:14 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:14.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:14 smithi086 ceph-mon[55252]: Reconfiguring osd.2 (monmap changed)... 2024-05-06T12:38:14.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:14 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:38:14.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:14 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:14.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:14 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.2 on smithi067 2024-05-06T12:38:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[62070]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[62070]: Reconfiguring osd.2 (monmap changed)... 2024-05-06T12:38:14.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.2 on smithi067 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[64635]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[64635]: Reconfiguring osd.2 (monmap changed)... 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:14.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:14 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.2 on smithi067 2024-05-06T12:38:15.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[62070]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:15.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[62070]: Reconfiguring osd.3 (monmap changed)... 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.3 on smithi067 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[64635]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[64635]: Reconfiguring osd.3 (monmap changed)... 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:38:15.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:15.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:15 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.3 on smithi067 2024-05-06T12:38:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:15 smithi086 ceph-mon[55252]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:15 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:15 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:15 smithi086 ceph-mon[55252]: Reconfiguring osd.3 (monmap changed)... 2024-05-06T12:38:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:15 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:38:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:15 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:16.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:15 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.3 on smithi067 2024-05-06T12:38:17.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:17.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:17.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[62070]: Reconfiguring rgw.foo.smithi067.iyetmn (monmap changed)... 2024-05-06T12:38:17.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[62070]: Reconfiguring daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[64635]: Reconfiguring rgw.foo.smithi067.iyetmn (monmap changed)... 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:17.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:17 smithi067 ceph-mon[64635]: Reconfiguring daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:38:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:17 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:17 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:17 smithi086 ceph-mon[55252]: Reconfiguring rgw.foo.smithi067.iyetmn (monmap changed)... 2024-05-06T12:38:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:17 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:17.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:17 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:17.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:17 smithi086 ceph-mon[55252]: Reconfiguring daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:38:18.698 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[62070]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[64635]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:18.699 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:18.869 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:38:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:38:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:38:18.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:18 smithi086 ceph-mon[55252]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:18.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:18.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:18.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:18.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:19.530 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:19 smithi067 ceph-mon[62070]: Reconfiguring rgw.smpl.smithi067.wpnjcz (monmap changed)... 2024-05-06T12:38:19.530 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:19 smithi067 ceph-mon[62070]: Reconfiguring daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:38:19.530 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:19.531 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:19 smithi067 ceph-mon[64635]: Reconfiguring rgw.smpl.smithi067.wpnjcz (monmap changed)... 2024-05-06T12:38:19.531 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:19 smithi067 ceph-mon[64635]: Reconfiguring daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:38:19.531 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:19.829 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:19 smithi086 ceph-mon[55252]: Reconfiguring rgw.smpl.smithi067.wpnjcz (monmap changed)... 2024-05-06T12:38:19.829 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:19 smithi086 ceph-mon[55252]: Reconfiguring daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:38:19.829 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:20.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:20 smithi086 ceph-mon[55252]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:20.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:20.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:20.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:20.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:20.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:20.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[64635]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:20.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:20.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:20.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:20.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:20.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[62070]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:20.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:20.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:20.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T12:38:20.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T12:38:20.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:21.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: Reconfiguring mon.b (monmap changed)... 2024-05-06T12:38:21.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: Reconfiguring daemon mon.b on smithi086 2024-05-06T12:38:21.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:21.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:21.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:21.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:38:21.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:38:21.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:21 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: Reconfiguring mon.b (monmap changed)... 2024-05-06T12:38:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: Reconfiguring daemon mon.b on smithi086 2024-05-06T12:38:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: Reconfiguring mon.b (monmap changed)... 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: Reconfiguring daemon mon.b on smithi086 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T12:38:21.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:21 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:22.570 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:22 smithi086 ceph-mon[55252]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:22.570 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:22 smithi086 ceph-mon[55252]: Reconfiguring mgr.x (monmap changed)... 2024-05-06T12:38:22.570 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:22 smithi086 ceph-mon[55252]: Reconfiguring daemon mgr.x on smithi086 2024-05-06T12:38:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:22 smithi067 ceph-mon[62070]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:22 smithi067 ceph-mon[62070]: Reconfiguring mgr.x (monmap changed)... 2024-05-06T12:38:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:22 smithi067 ceph-mon[62070]: Reconfiguring daemon mgr.x on smithi086 2024-05-06T12:38:22.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:22 smithi067 ceph-mon[64635]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:22.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:22 smithi067 ceph-mon[64635]: Reconfiguring mgr.x (monmap changed)... 2024-05-06T12:38:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:22 smithi067 ceph-mon[64635]: Reconfiguring daemon mgr.x on smithi086 2024-05-06T12:38:24.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:38:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:38:23] "GET /metrics HTTP/1.1" 200 38254 "" "Prometheus/2.33.4" 2024-05-06T12:38:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[62070]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[62070]: Reconfiguring osd.4 (monmap changed)... 2024-05-06T12:38:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:38:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.4 on smithi086 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[64635]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[64635]: Reconfiguring osd.4 (monmap changed)... 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:24.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:23 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.4 on smithi086 2024-05-06T12:38:24.227 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:23 smithi086 ceph-mon[55252]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:24.227 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:24.227 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:24.227 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:23 smithi086 ceph-mon[55252]: Reconfiguring osd.4 (monmap changed)... 2024-05-06T12:38:24.227 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:38:24.227 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:24.227 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:23 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.4 on smithi086 2024-05-06T12:38:25.842 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:25 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:25.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:25 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:25.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:25 smithi086 ceph-mon[55252]: Reconfiguring osd.5 (monmap changed)... 2024-05-06T12:38:25.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:25 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:38:25.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:25 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:25.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:25 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.5 on smithi086 2024-05-06T12:38:25.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:25 smithi086 ceph-mon[55252]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:25.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[62070]: Reconfiguring osd.5 (monmap changed)... 2024-05-06T12:38:25.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:38:25.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:25.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.5 on smithi086 2024-05-06T12:38:25.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[62070]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:25.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[64635]: Reconfiguring osd.5 (monmap changed)... 2024-05-06T12:38:25.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:38:25.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:25.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.5 on smithi086 2024-05-06T12:38:25.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:25 smithi067 ceph-mon[64635]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:27.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:27 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:27.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:27 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:27.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:27 smithi086 ceph-mon[55252]: Reconfiguring osd.6 (monmap changed)... 2024-05-06T12:38:27.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:27 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:38:27.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:27 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:27.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:27 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.6 on smithi086 2024-05-06T12:38:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[62070]: Reconfiguring osd.6 (monmap changed)... 2024-05-06T12:38:27.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.6 on smithi086 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[64635]: Reconfiguring osd.6 (monmap changed)... 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:27.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:27 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.6 on smithi086 2024-05-06T12:38:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[62070]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:38:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[64635]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:38:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:28.534 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:28 smithi086 ceph-mon[55252]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:28.534 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:28.534 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:28.534 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:38:28.534 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:28.605 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:38:28.836 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:38:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:38:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (2m) 48s ago 7m 17.1M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (2m) 29s ago 7m 43.8M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (105s) 48s ago 6m 49.8M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (96s) 29s ago 12m 449M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (2m) 48s ago 13m 493M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (81s) 48s ago 13m 39.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (39s) 29s ago 12m 19.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (55s) 48s ago 12m 18.4M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (2m) 48s ago 8m 16.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (2m) 29s ago 8m 16.0M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:38:29.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (11m) 48s ago 11m 52.6M 2736M 17.2.0 e1d6a67b021e e00ed115db64 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (11m) 48s ago 11m 53.8M 2736M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (11m) 48s ago 11m 47.7M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (10m) 48s ago 10m 51.5M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (10m) 29s ago 10m 53.3M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (9m) 29s ago 9m 50.3M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (9m) 29s ago 9m 51.2M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (9m) 29s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (101s) 29s ago 8m 58.2M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (6m) 48s ago 6m 74.7M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (6m) 29s ago 6m 72.4M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (6m) 48s ago 6m 74.2M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:38:29.290 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (6m) 29s ago 6m 72.5M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:38:29.481 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:29 smithi086 ceph-mon[55252]: Reconfiguring osd.7 (monmap changed)... 2024-05-06T12:38:29.482 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:29 smithi086 ceph-mon[55252]: Reconfiguring daemon osd.7 on smithi086 2024-05-06T12:38:29.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:29 smithi067 ceph-mon[62070]: Reconfiguring osd.7 (monmap changed)... 2024-05-06T12:38:29.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:29 smithi067 ceph-mon[62070]: Reconfiguring daemon osd.7 on smithi086 2024-05-06T12:38:29.704 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:29 smithi067 ceph-mon[64635]: Reconfiguring osd.7 (monmap changed)... 2024-05-06T12:38:29.704 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:29 smithi067 ceph-mon[64635]: Reconfiguring daemon osd.7 on smithi086 2024-05-06T12:38:29.704 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:38:29.704 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:38:29.704 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:38:29.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:38:29.704 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:38:29.704 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:38:29.705 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:38:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:38:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:38:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:38:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "5/23 daemons upgraded", 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mon daemons", 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:38:30.049 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:38:30.434 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:30.435 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:30 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/123396243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:30.464 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:30.465 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/123396243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:30.466 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:30 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/123396243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: Reconfiguring rgw.foo.smithi086.frwvbk (monmap changed)... 2024-05-06T12:38:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: Reconfiguring daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:38:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/194587062' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:38:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:31.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:31.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:31.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: Reconfiguring rgw.foo.smithi086.frwvbk (monmap changed)... 2024-05-06T12:38:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: Reconfiguring daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:38:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/194587062' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:38:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: Reconfiguring rgw.foo.smithi086.frwvbk (monmap changed)... 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: Reconfiguring daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/194587062' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:38:31.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:32 smithi086 ceph-mon[55252]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:32 smithi086 ceph-mon[55252]: Reconfiguring rgw.smpl.smithi086.myvasn (monmap changed)... 2024-05-06T12:38:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:32 smithi086 ceph-mon[55252]: Reconfiguring daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:38:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:32 smithi067 ceph-mon[62070]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:32 smithi067 ceph-mon[62070]: Reconfiguring rgw.smpl.smithi086.myvasn (monmap changed)... 2024-05-06T12:38:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:32 smithi067 ceph-mon[62070]: Reconfiguring daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:38:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:32 smithi067 ceph-mon[64635]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:32 smithi067 ceph-mon[64635]: Reconfiguring rgw.smpl.smithi086.myvasn (monmap changed)... 2024-05-06T12:38:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:32 smithi067 ceph-mon[64635]: Reconfiguring daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:38:33.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:38:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:38:33] "GET /metrics HTTP/1.1" 200 38254 "" "Prometheus/2.33.4" 2024-05-06T12:38:33.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: Upgrade: Setting container_image for all mon 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: Upgrade: Setting container_image for all crash 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[62070]: Upgrade: osd.0 is safe to restart 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: Upgrade: Setting container_image for all mon 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: Upgrade: Setting container_image for all crash 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T12:38:33.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:33 smithi067 ceph-mon[64635]: Upgrade: osd.0 is safe to restart 2024-05-06T12:38:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: Upgrade: Setting container_image for all mon 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: Upgrade: Setting container_image for all crash 2024-05-06T12:38:34.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:34.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T12:38:34.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T12:38:34.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:33 smithi086 ceph-mon[55252]: Upgrade: osd.0 is safe to restart 2024-05-06T12:38:34.971 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:34.971 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[64635]: Upgrade: Updating osd.0 2024-05-06T12:38:34.971 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[64635]: Deploying daemon osd.0 on smithi067 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[62070]: Upgrade: Updating osd.0 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:34.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:34 smithi067 ceph-mon[62070]: Deploying daemon osd.0 on smithi067 2024-05-06T12:38:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:34 smithi086 ceph-mon[55252]: Upgrade: Updating osd.0 2024-05-06T12:38:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T12:38:35.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:35.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:34 smithi086 ceph-mon[55252]: Deploying daemon osd.0 on smithi067 2024-05-06T12:38:36.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:36 smithi086 ceph-mon[55252]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:36.624 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:36 smithi067 ceph-mon[62070]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:36.625 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:36 smithi067 ceph-mon[64635]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:37.974 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:37 smithi067 systemd[1]: Stopping Ceph osd.0 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:38:38.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:37 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[29990]: 2024-05-06T12:38:37.971+0000 7f2e28ebf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:38:38.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:37 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[29990]: 2024-05-06T12:38:37.971+0000 7f2e28ebf700 -1 osd.0 105 *** Got signal Terminated *** 2024-05-06T12:38:38.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:37 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[29990]: 2024-05-06T12:38:37.971+0000 7f2e28ebf700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:38:38.660 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:38 smithi086 ceph-mon[55252]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:38.660 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:38 smithi086 ceph-mon[55252]: osd.0 marked itself down and dead 2024-05-06T12:38:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:38 smithi067 ceph-mon[62070]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:38 smithi067 ceph-mon[62070]: osd.0 marked itself down and dead 2024-05-06T12:38:38.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:38 smithi067 ceph-mon[64635]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:38:38.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:38 smithi067 ceph-mon[64635]: osd.0 marked itself down and dead 2024-05-06T12:38:39.116 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:38 smithi067 podman[67544]: 2024-05-06 12:38:38.837608413 +0000 UTC m=+0.960080783 container died e00ed115db64421fcc332957155ce2a1feb139438a0411450fdd4dea02388415 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vcs-type=git) 2024-05-06T12:38:39.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:38:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:38:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:38:39.395 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:39 smithi067 podman[67544]: 2024-05-06 12:38:39.214015054 +0000 UTC m=+1.336487423 container cleanup e00ed115db64421fcc332957155ce2a1feb139438a0411450fdd4dea02388415 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, release=754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64, distribution-scope=public, vcs-type=git, io.k8s.display-name=CentOS Stream 8, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-06T12:38:39.396 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:39 smithi067 bash[67544]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0 2024-05-06T12:38:39.662 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:39 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:38:39.662 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:39 smithi086 ceph-mon[55252]: osdmap e106: 8 total, 7 up, 8 in 2024-05-06T12:38:39.662 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:39 smithi086 ceph-mon[55252]: osdmap e107: 8 total, 7 up, 8 in 2024-05-06T12:38:39.713 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:39 smithi067 podman[67564]: 2024-05-06 12:38:39.490790034 +0000 UTC m=+0.652182885 container remove e00ed115db64421fcc332957155ce2a1feb139438a0411450fdd4dea02388415 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, architecture=x86_64, GIT_BRANCH=HEAD, vcs-type=git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, ceph=True, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux ) 2024-05-06T12:38:39.714 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:39 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:38:39.714 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:39 smithi067 ceph-mon[62070]: osdmap e106: 8 total, 7 up, 8 in 2024-05-06T12:38:39.714 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:39 smithi067 ceph-mon[62070]: osdmap e107: 8 total, 7 up, 8 in 2024-05-06T12:38:39.714 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:39 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:38:39.714 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:39 smithi067 ceph-mon[64635]: osdmap e106: 8 total, 7 up, 8 in 2024-05-06T12:38:39.714 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:39 smithi067 ceph-mon[64635]: osdmap e107: 8 total, 7 up, 8 in 2024-05-06T12:38:39.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:39 smithi067 podman[67657]: 2024-05-06 12:38:39.833505671 +0000 UTC m=+0.113183487 container create 07ecc4efb9319af968b371646566f3ba9cd36eea585705d02644c73d2fb9e299 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:38:39.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:39 smithi067 podman[67657]: 2024-05-06 12:38:39.738808998 +0000 UTC m=+0.018486831 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:38:40.378 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 podman[67657]: 2024-05-06 12:38:40.071588682 +0000 UTC m=+0.351266496 container init 07ecc4efb9319af968b371646566f3ba9cd36eea585705d02644c73d2fb9e299 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-06T12:38:40.378 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 podman[67657]: 2024-05-06 12:38:40.07508526 +0000 UTC m=+0.354763080 container start 07ecc4efb9319af968b371646566f3ba9cd36eea585705d02644c73d2fb9e299 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-06T12:38:40.378 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 podman[67657]: 2024-05-06 12:38:40.146475325 +0000 UTC m=+0.426153142 container attach 07ecc4efb9319af968b371646566f3ba9cd36eea585705d02644c73d2fb9e299 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-06T12:38:40.659 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 podman[67657]: 2024-05-06 12:38:40.377576956 +0000 UTC m=+0.657254772 container died 07ecc4efb9319af968b371646566f3ba9cd36eea585705d02644c73d2fb9e299 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-06T12:38:40.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:40 smithi067 ceph-mon[62070]: pgmap v44: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:38:40.660 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:40 smithi067 ceph-mon[64635]: pgmap v44: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:38:40.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:40 smithi086 ceph-mon[55252]: pgmap v44: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:38:41.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 podman[67683]: 2024-05-06 12:38:40.958957568 +0000 UTC m=+0.576423574 container remove 07ecc4efb9319af968b371646566f3ba9cd36eea585705d02644c73d2fb9e299 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-06T12:38:41.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.0.service: Deactivated successfully. 2024-05-06T12:38:41.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 systemd[1]: Stopped Ceph osd.0 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:38:41.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:40 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.0.service: Consumed 7.807s CPU time. 2024-05-06T12:38:41.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:41 smithi067 systemd[1]: Starting Ceph osd.0 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:38:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:41 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:41.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:41 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:41.726 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:41 smithi067 podman[67787]: 2024-05-06 12:38:41.418545176 +0000 UTC m=+0.113432281 container create a8276181e367ecc6586d2314488f84407258392ac57b1add81a270cf7dd627fe (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, ceph=True, CEPH_POINT_RELEASE=) 2024-05-06T12:38:41.726 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:41 smithi067 podman[67787]: 2024-05-06 12:38:41.323513498 +0000 UTC m=+0.018400619 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:38:41.726 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:41 smithi067 podman[67787]: 2024-05-06 12:38:41.629676456 +0000 UTC m=+0.324563561 container init a8276181e367ecc6586d2314488f84407258392ac57b1add81a270cf7dd627fe (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:38:41.726 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:41 smithi067 podman[67787]: 2024-05-06 12:38:41.633216808 +0000 UTC m=+0.328103919 container start a8276181e367ecc6586d2314488f84407258392ac57b1add81a270cf7dd627fe (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:38:41.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:41 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:42.096 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:41 smithi067 podman[67787]: 2024-05-06 12:38:41.734211143 +0000 UTC m=+0.429098249 container attach a8276181e367ecc6586d2314488f84407258392ac57b1add81a270cf7dd627fe (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate[67802]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate[67802]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 bash[67787]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 bash[67787]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 bash[67787]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 bash[67787]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 bash[67787]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 bash[67787]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 bash[67787]: --> ceph-volume raw activate successful for osd ID: 0 2024-05-06T12:38:42.405 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate[67802]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T12:38:42.406 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate[67802]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T12:38:42.406 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate[67802]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-05-06T12:38:42.406 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate[67802]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T12:38:42.406 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate[67802]: --> ceph-volume raw activate successful for osd ID: 0 2024-05-06T12:38:42.406 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 podman[67787]: 2024-05-06 12:38:42.1201392 +0000 UTC m=+0.815026301 container died a8276181e367ecc6586d2314488f84407258392ac57b1add81a270cf7dd627fe (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-06T12:38:42.701 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:42 smithi067 ceph-mon[62070]: pgmap v46: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:42.702 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:42 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 66/627 objects degraded (10.526%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T12:38:42.702 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:42 smithi067 ceph-mon[64635]: pgmap v46: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:42.702 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:42 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 66/627 objects degraded (10.526%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T12:38:42.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:42 smithi086 ceph-mon[55252]: pgmap v46: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:42.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:42 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 66/627 objects degraded (10.526%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T12:38:42.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 podman[67923]: 2024-05-06 12:38:42.700201645 +0000 UTC m=+0.574918782 container remove a8276181e367ecc6586d2314488f84407258392ac57b1add81a270cf7dd627fe (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-06T12:38:42.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 podman[67991]: 2024-05-06 12:38:42.820975677 +0000 UTC m=+0.018771931 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:38:42.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:42 smithi067 podman[67991]: 2024-05-06 12:38:42.917949147 +0000 UTC m=+0.115745427 container create e278b8d0f47ff549b36bca9049215e18e466224dd015652567bc89019d93c61a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-05-06T12:38:43.475 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:43 smithi067 podman[67991]: 2024-05-06 12:38:43.112164025 +0000 UTC m=+0.309960278 container init e278b8d0f47ff549b36bca9049215e18e466224dd015652567bc89019d93c61a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-06T12:38:43.475 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:43 smithi067 podman[67991]: 2024-05-06 12:38:43.115305519 +0000 UTC m=+0.313101766 container start e278b8d0f47ff549b36bca9049215e18e466224dd015652567bc89019d93c61a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-06T12:38:43.475 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:43 smithi067 ceph-osd[68010]: -- 172.21.15.67:0/797496586 <== mon.1 v2:172.21.15.67:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654d83a9680 con 0x5654d8373000 2024-05-06T12:38:43.475 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:43 smithi067 bash[67991]: e278b8d0f47ff549b36bca9049215e18e466224dd015652567bc89019d93c61a 2024-05-06T12:38:43.475 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:43 smithi067 systemd[1]: Started Ceph osd.0 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:38:43.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:38:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:38:43] "GET /metrics HTTP/1.1" 200 38251 "" "Prometheus/2.33.4" 2024-05-06T12:38:43.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[68006]: 2024-05-06T12:38:43.657+0000 7fb6e91307c0 -1 Falling back to public interface 2024-05-06T12:38:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:44 smithi067 ceph-mon[62070]: pgmap v47: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 66/627 objects degraded (10.526%) 2024-05-06T12:38:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:44 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:44.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:44 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:44 smithi067 ceph-mon[64635]: pgmap v47: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 66/627 objects degraded (10.526%) 2024-05-06T12:38:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:44 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:44.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:44 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:44.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:44 smithi086 ceph-mon[55252]: pgmap v47: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 66/627 objects degraded (10.526%) 2024-05-06T12:38:44.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:44 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:44.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:44 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:46.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:46 smithi086 ceph-mon[55252]: pgmap v48: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:46.647 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:46 smithi067 ceph-mon[62070]: pgmap v48: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:46.648 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:46 smithi067 ceph-mon[64635]: pgmap v48: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:46.648 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[68006]: 2024-05-06T12:38:46.365+0000 7fb6e91307c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-05-06T12:38:46.648 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[68006]: 2024-05-06T12:38:46.392+0000 7fb6e91307c0 -1 osd.0 105 log_to_monitors true 2024-05-06T12:38:47.530 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:47 smithi067 ceph-mon[62070]: from='osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:38:47.531 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:47 smithi067 ceph-mon[62070]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:38:47.531 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:47 smithi067 ceph-mon[64635]: from='osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:38:47.531 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:47 smithi067 ceph-mon[64635]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:38:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:47 smithi086 ceph-mon[55252]: from='osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:38:47.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:47 smithi086 ceph-mon[55252]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T12:38:47.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:38:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[68006]: 2024-05-06T12:38:47.743+0000 7fb6d657f700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: pgmap v49: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 493 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: osdmap e108: 8 total, 7 up, 8 in 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: from='osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: from='osd.0 ' entity='osd.0' 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:48.579 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: pgmap v49: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 493 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: osdmap e108: 8 total, 7 up, 8 in 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: from='osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: from='osd.0 ' entity='osd.0' 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:48.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: pgmap v49: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 493 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T12:38:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: osdmap e108: 8 total, 7 up, 8 in 2024-05-06T12:38:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: from='osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:38:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:38:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: from='osd.0 ' entity='osd.0' 2024-05-06T12:38:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:49.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:38:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:38:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:38:49.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 440 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:49.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:49.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:49.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:38:49.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760] boot 2024-05-06T12:38:49.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: osdmap e109: 8 total, 8 up, 8 in 2024-05-06T12:38:49.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 440 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760] boot 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: osdmap e109: 8 total, 8 up, 8 in 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:49.921 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 440 B/s rd, 0 op/s; 66/627 objects degraded (10.526%) 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: osd.0 [v2:172.21.15.67:6802/77185760,v1:172.21.15.67:6803/77185760] boot 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: osdmap e109: 8 total, 8 up, 8 in 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:50.062 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:50.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:50 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:50.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:50 smithi067 ceph-mon[62070]: osdmap e110: 8 total, 8 up, 8 in 2024-05-06T12:38:50.949 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:50 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:50.949 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:50 smithi067 ceph-mon[64635]: osdmap e110: 8 total, 8 up, 8 in 2024-05-06T12:38:51.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:50 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:38:51.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:50 smithi086 ceph-mon[55252]: osdmap e110: 8 total, 8 up, 8 in 2024-05-06T12:38:52.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:51 smithi086 ceph-mon[55252]: pgmap v54: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-05-06T12:38:52.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:51 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T12:38:52.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:51 smithi067 ceph-mon[64635]: pgmap v54: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-05-06T12:38:52.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:51 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T12:38:52.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:51 smithi067 ceph-mon[62070]: pgmap v54: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-05-06T12:38:52.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:51 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T12:38:52.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:38:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:38:52.913+0000 7f337218f700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:38:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-05-06T12:38:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:38:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:38:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:52 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-05-06T12:38:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:52 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:38:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:38:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:38:53.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:38:54.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: pgmap v55: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T12:38:54.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:38:54.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:38:54.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:38:54.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:53 smithi086 ceph-mon[55252]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:38:54.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:38:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:38:53] "GET /metrics HTTP/1.1" 200 38237 "" "Prometheus/2.33.4" 2024-05-06T12:38:54.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: pgmap v55: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T12:38:54.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:38:54.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[64635]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: pgmap v55: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:38:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:53 smithi067 ceph-mon[62070]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:38:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:55 smithi086 ceph-mon[55252]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:56.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:55 smithi067 ceph-mon[62070]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:56.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:55 smithi067 ceph-mon[64635]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:38:58.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:57 smithi086 ceph-mon[55252]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:38:58.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:57 smithi067 ceph-mon[62070]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:38:58.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:57 smithi067 ceph-mon[64635]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:38:59.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:38:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:38:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-05-06T12:38:59.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:38:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:38:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:39:00.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:38:59 smithi086 ceph-mon[55252]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-05-06T12:39:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:38:59 smithi067 ceph-mon[62070]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-05-06T12:39:00.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:38:59 smithi067 ceph-mon[64635]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-05-06T12:39:00.839 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:39:01.122 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:00 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:00 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:01.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:00 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (3m) 13s ago 8m 24.9M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (2m) 61s ago 7m 43.8M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (2m) 13s ago 7m 52.6M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (2m) 61s ago 12m 449M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (2m) 13s ago 14m 500M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (113s) 13s ago 14m 44.7M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (71s) 61s ago 12m 19.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:39:01.517 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (87s) 13s ago 13m 35.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (3m) 13s ago 8m 17.3M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (2m) 61s ago 8m 16.0M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (18s) 13s ago 12m 12.7M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (11m) 13s ago 11m 53.4M 2736M 17.2.0 e1d6a67b021e 0326bc06fdf8 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (11m) 13s ago 11m 48.5M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (11m) 13s ago 11m 52.6M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (10m) 61s ago 10m 53.3M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (10m) 61s ago 10m 50.3M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (10m) 61s ago 10m 51.2M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (9m) 61s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (2m) 61s ago 8m 58.2M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (7m) 13s ago 7m 75.0M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:39:01.518 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (7m) 61s ago 7m 72.4M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:39:01.519 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (7m) 13s ago 7m 74.5M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:39:01.519 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (7m) 61s ago 7m 72.5M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:39:01.925 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:39:01.925 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:39:01.925 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 6 2024-05-06T12:39:01.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:39:01.927 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:39:02.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:01 smithi086 ceph-mon[55252]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T12:39:02.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:01 smithi086 ceph-mon[55252]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:01 smithi086 ceph-mon[55252]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:01 smithi067 ceph-mon[64635]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T12:39:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:01 smithi067 ceph-mon[64635]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:01 smithi067 ceph-mon[64635]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:01 smithi067 ceph-mon[62070]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T12:39:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:01 smithi067 ceph-mon[62070]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:01 smithi067 ceph-mon[62070]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.275 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:39:02.275 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:39:02.275 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "6/23 daemons upgraded", 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:39:02.276 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:39:02.689 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:39:02.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[62070]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/4082282578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:02.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[62070]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3090558704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:39:02.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[64635]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/4082282578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:02.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[64635]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:02.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:02 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3090558704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:39:03.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:02 smithi086 ceph-mon[55252]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:03.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:02 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/4082282578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:03.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:02 smithi086 ceph-mon[55252]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:03.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:02 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3090558704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:39:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:03 smithi086 ceph-mon[55252]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:04.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:39:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:39:03] "GET /metrics HTTP/1.1" 200 38237 "" "Prometheus/2.33.4" 2024-05-06T12:39:04.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[62070]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:04.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:04.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:04.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[64635]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:05 smithi086 ceph-mon[55252]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:06.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:05 smithi067 ceph-mon[62070]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:06.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:05 smithi067 ceph-mon[64635]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:08.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:07 smithi086 ceph-mon[55252]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:07 smithi067 ceph-mon[62070]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:08.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:07 smithi067 ceph-mon[64635]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:09.020 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:39:09.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:08 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:39:09.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:08 smithi086 ceph-mon[55252]: Upgrade: osd.1 is safe to restart 2024-05-06T12:39:09.021 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:08.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-05-06T12:39:09.021 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:39:09.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:39:09.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:08 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:39:09.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:08 smithi067 ceph-mon[62070]: Upgrade: osd.1 is safe to restart 2024-05-06T12:39:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:39:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:08 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T12:39:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:08 smithi067 ceph-mon[64635]: Upgrade: osd.1 is safe to restart 2024-05-06T12:39:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[64635]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:39:09.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[62070]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:09.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:39:09.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:09 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:10.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:09 smithi086 ceph-mon[55252]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:10.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:09 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:10.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:09 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T12:39:10.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:09 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:11.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:10 smithi086 ceph-mon[55252]: Upgrade: Updating osd.1 2024-05-06T12:39:11.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:10 smithi086 ceph-mon[55252]: Deploying daemon osd.1 on smithi067 2024-05-06T12:39:11.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:10 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:10 smithi067 ceph-mon[62070]: Upgrade: Updating osd.1 2024-05-06T12:39:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:10 smithi067 ceph-mon[62070]: Deploying daemon osd.1 on smithi067 2024-05-06T12:39:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:10 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:11.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:10 smithi067 ceph-mon[64635]: Upgrade: Updating osd.1 2024-05-06T12:39:11.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:10 smithi067 ceph-mon[64635]: Deploying daemon osd.1 on smithi067 2024-05-06T12:39:11.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:10 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:11.840 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:11 smithi067 ceph-mon[62070]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:12.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:11 smithi086 ceph-mon[55252]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:12.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:11 smithi067 ceph-mon[64635]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:13.225 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:12 smithi067 systemd[1]: Stopping Ceph osd.1 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:39:13.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[32953]: 2024-05-06T12:39:13.241+0000 7fbec712c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:39:13.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[32953]: 2024-05-06T12:39:13.241+0000 7fbec712c700 -1 osd.1 110 *** Got signal Terminated *** 2024-05-06T12:39:13.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[32953]: 2024-05-06T12:39:13.241+0000 7fbec712c700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:39:14.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:13 smithi086 ceph-mon[55252]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:14.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:13 smithi086 ceph-mon[55252]: osd.1 marked itself down and dead 2024-05-06T12:39:14.213 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:39:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:39:13] "GET /metrics HTTP/1.1" 200 38395 "" "Prometheus/2.33.4" 2024-05-06T12:39:14.213 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:13 smithi067 ceph-mon[62070]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:14.213 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:13 smithi067 ceph-mon[62070]: osd.1 marked itself down and dead 2024-05-06T12:39:14.213 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:13 smithi067 ceph-mon[64635]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:14.213 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:13 smithi067 ceph-mon[64635]: osd.1 marked itself down and dead 2024-05-06T12:39:14.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:14 smithi067 podman[70020]: 2024-05-06 12:39:14.212412418 +0000 UTC m=+1.081030058 container died 0326bc06fdf8f4a941377cd1fb9fcbde6d8fee47be90b750139dc2bb36b3fba5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., ceph=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, RELEASE=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870) 2024-05-06T12:39:14.841 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:14 smithi067 podman[70020]: 2024-05-06 12:39:14.591388671 +0000 UTC m=+1.460006301 container cleanup 0326bc06fdf8f4a941377cd1fb9fcbde6d8fee47be90b750139dc2bb36b3fba5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, distribution-scope=public, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , ceph=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-06T12:39:14.841 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:14 smithi067 bash[70020]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1 2024-05-06T12:39:15.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:14 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:39:15.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:14 smithi086 ceph-mon[55252]: osdmap e111: 8 total, 7 up, 8 in 2024-05-06T12:39:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:14 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:39:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:14 smithi067 ceph-mon[62070]: osdmap e111: 8 total, 7 up, 8 in 2024-05-06T12:39:15.222 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:14 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:39:15.222 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:14 smithi067 ceph-mon[64635]: osdmap e111: 8 total, 7 up, 8 in 2024-05-06T12:39:15.222 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:14 smithi067 podman[70038]: 2024-05-06 12:39:14.884437484 +0000 UTC m=+0.674958063 container remove 0326bc06fdf8f4a941377cd1fb9fcbde6d8fee47be90b750139dc2bb36b3fba5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, distribution-scope=public, version=8, release=754, io.openshift.tags=base centos centos-stream, ceph=True, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-06T12:39:15.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:15 smithi067 podman[70129]: 2024-05-06 12:39:15.123844106 +0000 UTC m=+0.018531860 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:39:15.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:15 smithi067 podman[70129]: 2024-05-06 12:39:15.244394039 +0000 UTC m=+0.139081781 container create 08e01b9ed5689580419efc70b5e1afa8f0839812250bda710804dfbe65193924 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430) 2024-05-06T12:39:15.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:15 smithi067 podman[70129]: 2024-05-06 12:39:15.463471157 +0000 UTC m=+0.358158900 container init 08e01b9ed5689580419efc70b5e1afa8f0839812250bda710804dfbe65193924 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-06T12:39:15.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:15 smithi067 podman[70129]: 2024-05-06 12:39:15.466886731 +0000 UTC m=+0.361574474 container start 08e01b9ed5689580419efc70b5e1afa8f0839812250bda710804dfbe65193924 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=) 2024-05-06T12:39:15.842 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:15 smithi067 podman[70129]: 2024-05-06 12:39:15.563694564 +0000 UTC m=+0.458382314 container attach 08e01b9ed5689580419efc70b5e1afa8f0839812250bda710804dfbe65193924 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2) 2024-05-06T12:39:15.842 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:15 smithi067 podman[70129]: 2024-05-06 12:39:15.772913705 +0000 UTC m=+0.667601448 container died 08e01b9ed5689580419efc70b5e1afa8f0839812250bda710804dfbe65193924 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-06T12:39:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:15 smithi086 ceph-mon[55252]: pgmap v67: 161 pgs: 57 peering, 1 stale+active+clean, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:15 smithi086 ceph-mon[55252]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-05-06T12:39:16.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:15 smithi086 ceph-mon[55252]: osdmap e112: 8 total, 7 up, 8 in 2024-05-06T12:39:16.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:15 smithi067 ceph-mon[62070]: pgmap v67: 161 pgs: 57 peering, 1 stale+active+clean, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:16.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:15 smithi067 ceph-mon[62070]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-05-06T12:39:16.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:15 smithi067 ceph-mon[62070]: osdmap e112: 8 total, 7 up, 8 in 2024-05-06T12:39:16.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:15 smithi067 ceph-mon[64635]: pgmap v67: 161 pgs: 57 peering, 1 stale+active+clean, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:16.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:15 smithi067 ceph-mon[64635]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-05-06T12:39:16.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:15 smithi067 ceph-mon[64635]: osdmap e112: 8 total, 7 up, 8 in 2024-05-06T12:39:16.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:16 smithi067 podman[70154]: 2024-05-06 12:39:16.317388423 +0000 UTC m=+0.535616321 container remove 08e01b9ed5689580419efc70b5e1afa8f0839812250bda710804dfbe65193924 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:39:16.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:16 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.1.service: Deactivated successfully. 2024-05-06T12:39:16.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:16 smithi067 systemd[1]: Stopped Ceph osd.1 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:39:16.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:16 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.1.service: Consumed 8.407s CPU time. 2024-05-06T12:39:16.725 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:16 smithi067 systemd[1]: Starting Ceph osd.1 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:39:17.031 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:16 smithi067 podman[70258]: 2024-05-06 12:39:16.672543823 +0000 UTC m=+0.018805207 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:39:17.031 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:16 smithi067 podman[70258]: 2024-05-06 12:39:16.802624139 +0000 UTC m=+0.148885508 container create 06ceba32fe5c2504fc51d673ccaf8526fe916897e14ab5135103b9c2c818a6f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-06T12:39:17.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 podman[70258]: 2024-05-06 12:39:17.03067992 +0000 UTC m=+0.376941299 container init 06ceba32fe5c2504fc51d673ccaf8526fe916897e14ab5135103b9c2c818a6f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-05-06T12:39:17.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 podman[70258]: 2024-05-06 12:39:17.034221427 +0000 UTC m=+0.380482805 container start 06ceba32fe5c2504fc51d673ccaf8526fe916897e14ab5135103b9c2c818a6f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0) 2024-05-06T12:39:17.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 podman[70258]: 2024-05-06 12:39:17.138620511 +0000 UTC m=+0.484881879 container attach 06ceba32fe5c2504fc51d673ccaf8526fe916897e14ab5135103b9c2c818a6f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate[70273]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate[70273]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 bash[70258]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 bash[70258]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 bash[70258]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 bash[70258]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 bash[70258]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-05-06T12:39:17.807 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 bash[70258]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T12:39:17.808 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 bash[70258]: --> ceph-volume raw activate successful for osd ID: 1 2024-05-06T12:39:17.808 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate[70273]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T12:39:17.808 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate[70273]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T12:39:17.808 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate[70273]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-05-06T12:39:17.808 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate[70273]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T12:39:17.808 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate[70273]: --> ceph-volume raw activate successful for osd ID: 1 2024-05-06T12:39:17.808 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:17 smithi067 podman[70258]: 2024-05-06 12:39:17.518131602 +0000 UTC m=+0.864392988 container died 06ceba32fe5c2504fc51d673ccaf8526fe916897e14ab5135103b9c2c818a6f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-06T12:39:18.109 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:17 smithi067 ceph-mon[62070]: pgmap v69: 161 pgs: 57 peering, 1 stale+active+clean, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:18.110 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:17 smithi067 ceph-mon[64635]: pgmap v69: 161 pgs: 57 peering, 1 stale+active+clean, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:18.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:17 smithi086 ceph-mon[55252]: pgmap v69: 161 pgs: 57 peering, 1 stale+active+clean, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:18.454 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 podman[70394]: 2024-05-06 12:39:18.10920451 +0000 UTC m=+0.585463985 container remove 06ceba32fe5c2504fc51d673ccaf8526fe916897e14ab5135103b9c2c818a6f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430) 2024-05-06T12:39:18.454 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 podman[70459]: 2024-05-06 12:39:18.237850295 +0000 UTC m=+0.019087965 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:39:18.455 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 podman[70459]: 2024-05-06 12:39:18.3522431 +0000 UTC m=+0.133480799 container create b2661afeacd0cfce448b5768871a0360025ef718f9fb584c05d644860f10b0b8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-06T12:39:18.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 podman[70459]: 2024-05-06 12:39:18.563365192 +0000 UTC m=+0.344602868 container init b2661afeacd0cfce448b5768871a0360025ef718f9fb584c05d644860f10b0b8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:39:18.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 podman[70459]: 2024-05-06 12:39:18.566927115 +0000 UTC m=+0.348164773 container start b2661afeacd0cfce448b5768871a0360025ef718f9fb584c05d644860f10b0b8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-06T12:39:18.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 ceph-osd[70478]: -- 172.21.15.67:0/571010479 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b9d8c1680 con 0x558b9d88b000 2024-05-06T12:39:18.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 bash[70459]: b2661afeacd0cfce448b5768871a0360025ef718f9fb584c05d644860f10b0b8 2024-05-06T12:39:18.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:18 smithi067 systemd[1]: Started Ceph osd.1 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:39:19.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:39:19.475 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:19 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[70474]: 2024-05-06T12:39:19.109+0000 7fef30e497c0 -1 Falling back to public interface 2024-05-06T12:39:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[62070]: pgmap v70: 161 pgs: 2 active+undersized+degraded, 57 peering, 4 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 7/627 objects degraded (1.116%) 2024-05-06T12:39:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:19.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 7/627 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:19.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[64635]: pgmap v70: 161 pgs: 2 active+undersized+degraded, 57 peering, 4 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 7/627 objects degraded (1.116%) 2024-05-06T12:39:19.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:19.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:19.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:19.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:19.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:19 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 7/627 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:20.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:19 smithi086 ceph-mon[55252]: pgmap v70: 161 pgs: 2 active+undersized+degraded, 57 peering, 4 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 7/627 objects degraded (1.116%) 2024-05-06T12:39:20.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:20.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:20.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:20.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:20.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:19 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 7/627 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:20.761 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:20 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:20.762 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:20 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:21.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:20 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:21.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:21 smithi067 ceph-mon[62070]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-05-06T12:39:21.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:21 smithi067 ceph-mon[62070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-05-06T12:39:21.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:21 smithi067 ceph-mon[64635]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-05-06T12:39:21.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:21 smithi067 ceph-mon[64635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-05-06T12:39:21.976 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:21 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[70474]: 2024-05-06T12:39:21.820+0000 7fef30e497c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-05-06T12:39:21.976 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:21 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[70474]: 2024-05-06T12:39:21.848+0000 7fef30e497c0 -1 osd.1 110 log_to_monitors true 2024-05-06T12:39:22.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:21 smithi086 ceph-mon[55252]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-05-06T12:39:22.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:21 smithi086 ceph-mon[55252]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-05-06T12:39:22.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:22 smithi067 ceph-mon[62070]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:39:22.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:22 smithi067 ceph-mon[64635]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:39:23.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:22 smithi086 ceph-mon[55252]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T12:39:23.292 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:39:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[70474]: 2024-05-06T12:39:23.209+0000 7fef1e298700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:39:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:23 smithi086 ceph-mon[55252]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-05-06T12:39:24.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:23 smithi086 ceph-mon[55252]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T12:39:24.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:23 smithi086 ceph-mon[55252]: osdmap e113: 8 total, 7 up, 8 in 2024-05-06T12:39:24.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:23 smithi086 ceph-mon[55252]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:24.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:23 smithi086 ceph-mon[55252]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' 2024-05-06T12:39:24.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:24.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:24.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:39:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:39:23] "GET /metrics HTTP/1.1" 200 38460 "" "Prometheus/2.33.4" 2024-05-06T12:39:24.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[62070]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-05-06T12:39:24.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[62070]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T12:39:24.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[62070]: osdmap e113: 8 total, 7 up, 8 in 2024-05-06T12:39:24.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[62070]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:24.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[62070]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' 2024-05-06T12:39:24.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[64635]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[64635]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[64635]: osdmap e113: 8 total, 7 up, 8 in 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[64635]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[64635]: from='osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317]' entity='osd.1' 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:24.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:25.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:24 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:39:25.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:24 smithi086 ceph-mon[55252]: osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317] boot 2024-05-06T12:39:25.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:24 smithi086 ceph-mon[55252]: osdmap e114: 8 total, 8 up, 8 in 2024-05-06T12:39:25.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:24 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:39:25.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:24 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:25.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:24 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:25.218 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:39:25.218 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[62070]: osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317] boot 2024-05-06T12:39:25.218 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[62070]: osdmap e114: 8 total, 8 up, 8 in 2024-05-06T12:39:25.218 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:39:25.218 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:25.218 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:25.218 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:39:25.219 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[64635]: osd.1 [v2:172.21.15.67:6810/1491898317,v1:172.21.15.67:6811/1491898317] boot 2024-05-06T12:39:25.219 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[64635]: osdmap e114: 8 total, 8 up, 8 in 2024-05-06T12:39:25.219 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T12:39:25.219 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:25.219 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:24 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:26.427 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:26 smithi067 ceph-mon[62070]: pgmap v75: 161 pgs: 37 peering, 8 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/627 objects degraded (6.380%) 2024-05-06T12:39:26.427 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:26 smithi067 ceph-mon[62070]: osdmap e115: 8 total, 8 up, 8 in 2024-05-06T12:39:26.427 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:26 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 40/627 objects degraded (6.380%), 8 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:26.427 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:26 smithi067 ceph-mon[64635]: pgmap v75: 161 pgs: 37 peering, 8 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/627 objects degraded (6.380%) 2024-05-06T12:39:26.427 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:26 smithi067 ceph-mon[64635]: osdmap e115: 8 total, 8 up, 8 in 2024-05-06T12:39:26.427 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:26 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 40/627 objects degraded (6.380%), 8 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:26.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:26 smithi086 ceph-mon[55252]: pgmap v75: 161 pgs: 37 peering, 8 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/627 objects degraded (6.380%) 2024-05-06T12:39:26.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:26 smithi086 ceph-mon[55252]: osdmap e115: 8 total, 8 up, 8 in 2024-05-06T12:39:26.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:26 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 40/627 objects degraded (6.380%), 8 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:28.213 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:39:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:39:28.057+0000 7f337218f700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: pgmap v77: 161 pgs: 37 peering, 8 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.470 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: pgmap v77: 161 pgs: 37 peering, 8 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: pgmap v77: 161 pgs: 37 peering, 8 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-05-06T12:39:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:39:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:39:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:29.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:28.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-05-06T12:39:29.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:39:29.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:29 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:29.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:29 smithi067 ceph-mon[62070]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-06T12:39:29.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:29 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:29.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:29 smithi067 ceph-mon[64635]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-06T12:39:29.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:29 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:29.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:29 smithi086 ceph-mon[55252]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-06T12:39:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:30 smithi067 ceph-mon[62070]: pgmap v78: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:30 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 8 pgs degraded) 2024-05-06T12:39:30.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:30 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:39:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:30 smithi067 ceph-mon[64635]: pgmap v78: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:30 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 8 pgs degraded) 2024-05-06T12:39:30.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:30 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:39:30.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:30 smithi086 ceph-mon[55252]: pgmap v78: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:30.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:30 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 8 pgs degraded) 2024-05-06T12:39:30.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:30 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:39:31.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:31 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:31 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:31.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:31 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:32.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:32 smithi086 ceph-mon[55252]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:39:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:32 smithi067 ceph-mon[62070]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:39:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:32 smithi067 ceph-mon[64635]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:39:33.069 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:39:33.761 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:39:33.761 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (3m) 10s ago 8m 24.9M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (3m) 93s ago 8m 43.8M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (2m) 10s ago 7m 52.7M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (2m) 93s ago 13m 449M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (3m) 10s ago 14m 505M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (2m) 10s ago 14m 46.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (103s) 93s ago 13m 19.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (119s) 10s ago 13m 37.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (3m) 10s ago 9m 19.8M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (3m) 93s ago 9m 16.0M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (50s) 10s ago 12m 49.7M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (15s) 10s ago 12m 12.1M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (12m) 10s ago 12m 50.5M 2736M 17.2.0 e1d6a67b021e c322a3b11e43 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (11m) 10s ago 11m 53.2M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:39:33.762 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (11m) 93s ago 11m 53.3M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (10m) 93s ago 10m 50.3M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (10m) 93s ago 10m 51.2M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (10m) 93s ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (2m) 93s ago 9m 58.2M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (7m) 10s ago 7m 75.2M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (7m) 93s ago 7m 72.4M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (7m) 10s ago 7m 74.7M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:39:33.763 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (7m) 93s ago 7m 72.5M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:39:34.176 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:39:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:39:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:39:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-05-06T12:39:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 7 2024-05-06T12:39:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:39:34.178 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:39:34.178 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:39:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:39:33] "GET /metrics HTTP/1.1" 200 38460 "" "Prometheus/2.33.4" 2024-05-06T12:39:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[62070]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[62070]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[62070]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/4227800733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:34.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[64635]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[64635]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[64635]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:34 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/4227800733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:34.529 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:39:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "7/23 daemons upgraded", 2024-05-06T12:39:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:39:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:39:34.531 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:39:34.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:34 smithi086 ceph-mon[55252]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:34.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:34 smithi086 ceph-mon[55252]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:34.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:34 smithi086 ceph-mon[55252]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:34.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:34.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:34.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:34 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/4227800733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:39:34.937 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:39:35.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:35 smithi086 ceph-mon[55252]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:35.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:35 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/70967328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:39:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:35 smithi067 ceph-mon[62070]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:35.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:35 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/70967328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:39:35.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:35 smithi067 ceph-mon[64635]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:35.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:35 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/70967328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:39:36.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:36 smithi086 ceph-mon[55252]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:36.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:36 smithi086 ceph-mon[55252]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:39:36.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:36 smithi067 ceph-mon[62070]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:36.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:36 smithi067 ceph-mon[62070]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:39:36.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:36 smithi067 ceph-mon[64635]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:39:36.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:36 smithi067 ceph-mon[64635]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:39:38.531 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:38 smithi067 ceph-mon[64635]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-05-06T12:39:38.532 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:38 smithi067 ceph-mon[62070]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-05-06T12:39:38.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:38 smithi086 ceph-mon[55252]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-05-06T12:39:39.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-05-06T12:39:39.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:39:40.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:39 smithi086 ceph-mon[55252]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:40.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:39 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:40.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:39 smithi067 ceph-mon[64635]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:40.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:39 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:39 smithi067 ceph-mon[62070]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:39 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:41.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:40 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:40 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:40 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:42.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:41 smithi086 ceph-mon[55252]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:41 smithi067 ceph-mon[62070]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:42.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:41 smithi067 ceph-mon[64635]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:43.952 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:39:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:39:43] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.33.4" 2024-05-06T12:39:43.952 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[64635]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:43.952 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:43.953 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:43.953 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[64635]: Upgrade: osd.2 is safe to restart 2024-05-06T12:39:43.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[62070]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:43.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:43.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:43.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:43 smithi067 ceph-mon[62070]: Upgrade: osd.2 is safe to restart 2024-05-06T12:39:44.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:43 smithi086 ceph-mon[55252]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:44.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:43 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:44.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:43 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T12:39:44.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:43 smithi086 ceph-mon[55252]: Upgrade: osd.2 is safe to restart 2024-05-06T12:39:45.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[62070]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:45.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[62070]: Upgrade: Updating osd.2 2024-05-06T12:39:45.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[62070]: Deploying daemon osd.2 on smithi067 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[64635]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[64635]: Upgrade: Updating osd.2 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:45.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:45 smithi067 ceph-mon[64635]: Deploying daemon osd.2 on smithi067 2024-05-06T12:39:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:45 smithi086 ceph-mon[55252]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:39:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:45 smithi086 ceph-mon[55252]: Upgrade: Updating osd.2 2024-05-06T12:39:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:45 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:45 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T12:39:46.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:45 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:39:46.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:45 smithi086 ceph-mon[55252]: Deploying daemon osd.2 on smithi067 2024-05-06T12:39:47.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:47 smithi067 ceph-mon[62070]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:47.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:47 smithi067 ceph-mon[64635]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:48.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:47 smithi086 ceph-mon[55252]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:48.293 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:48 smithi067 systemd[1]: Stopping Ceph osd.2 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:39:48.725 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[35890]: 2024-05-06T12:39:48.290+0000 7f0fadb70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:39:48.725 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[35890]: 2024-05-06T12:39:48.290+0000 7f0fadb70700 -1 osd.2 115 *** Got signal Terminated *** 2024-05-06T12:39:48.725 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[35890]: 2024-05-06T12:39:48.290+0000 7f0fadb70700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:39:49.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:48 smithi086 ceph-mon[55252]: osd.2 marked itself down and dead 2024-05-06T12:39:49.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:49.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:39:49.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:48 smithi067 ceph-mon[62070]: osd.2 marked itself down and dead 2024-05-06T12:39:49.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:49.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:48 smithi067 ceph-mon[64635]: osd.2 marked itself down and dead 2024-05-06T12:39:49.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:39:49.577 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:49 smithi067 podman[72493]: 2024-05-06 12:39:49.270324473 +0000 UTC m=+1.104991750 container died c322a3b11e431948880ac62ab3b1820629a3a5d6be031c0b39c2e98daaae9a27 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, ceph=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vcs-type=git, name=centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, distribution-scope=public, release=754, io.k8s.display-name=CentOS Stream 8) 2024-05-06T12:39:49.927 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:49 smithi067 ceph-mon[62070]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:49.927 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:49 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:39:49.927 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:49 smithi067 ceph-mon[62070]: osdmap e116: 8 total, 7 up, 8 in 2024-05-06T12:39:49.928 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:49 smithi067 ceph-mon[64635]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:49.928 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:49 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:39:49.928 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:49 smithi067 ceph-mon[64635]: osdmap e116: 8 total, 7 up, 8 in 2024-05-06T12:39:49.928 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:49 smithi067 podman[72493]: 2024-05-06 12:39:49.691788082 +0000 UTC m=+1.526455354 container cleanup c322a3b11e431948880ac62ab3b1820629a3a5d6be031c0b39c2e98daaae9a27 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, release=754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-06T12:39:49.928 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:49 smithi067 bash[72493]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2 2024-05-06T12:39:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:49 smithi086 ceph-mon[55252]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:39:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:49 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:39:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:49 smithi086 ceph-mon[55252]: osdmap e116: 8 total, 7 up, 8 in 2024-05-06T12:39:50.225 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:49 smithi067 podman[72511]: 2024-05-06 12:39:49.926700479 +0000 UTC m=+0.650289939 container remove c322a3b11e431948880ac62ab3b1820629a3a5d6be031c0b39c2e98daaae9a27 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, RELEASE=HEAD, architecture=x86_64, io.openshift.expose-services=, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, release=754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, version=8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-06T12:39:50.672 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:50 smithi067 podman[72606]: 2024-05-06 12:39:50.268170855 +0000 UTC m=+0.018767128 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:39:50.673 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:50 smithi067 podman[72606]: 2024-05-06 12:39:50.386249364 +0000 UTC m=+0.136845685 container create 7c9a95a58fecb54e4aa05032fbc41501bf9749448f713da5f3e1baec3865033f (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:39:50.673 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:50 smithi067 podman[72606]: 2024-05-06 12:39:50.597259614 +0000 UTC m=+0.347855896 container init 7c9a95a58fecb54e4aa05032fbc41501bf9749448f713da5f3e1baec3865033f (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:39:50.673 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:50 smithi067 podman[72606]: 2024-05-06 12:39:50.60079029 +0000 UTC m=+0.351386569 container start 7c9a95a58fecb54e4aa05032fbc41501bf9749448f713da5f3e1baec3865033f (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-05-06T12:39:50.673 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:50 smithi067 podman[72606]: 2024-05-06 12:39:50.671903092 +0000 UTC m=+0.422499367 container attach 7c9a95a58fecb54e4aa05032fbc41501bf9749448f713da5f3e1baec3865033f (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:39:50.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:50 smithi067 ceph-mon[62070]: osdmap e117: 8 total, 7 up, 8 in 2024-05-06T12:39:50.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:50 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:50.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:50 smithi067 ceph-mon[64635]: osdmap e117: 8 total, 7 up, 8 in 2024-05-06T12:39:50.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:50 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:50.976 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:50 smithi067 podman[72606]: 2024-05-06 12:39:50.901011251 +0000 UTC m=+0.651607524 container died 7c9a95a58fecb54e4aa05032fbc41501bf9749448f713da5f3e1baec3865033f (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-70ce953) 2024-05-06T12:39:51.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:50 smithi086 ceph-mon[55252]: osdmap e117: 8 total, 7 up, 8 in 2024-05-06T12:39:51.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:50 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:39:51.766 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:51 smithi067 podman[72632]: 2024-05-06 12:39:51.484166432 +0000 UTC m=+0.576713796 container remove 7c9a95a58fecb54e4aa05032fbc41501bf9749448f713da5f3e1baec3865033f (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-05-06T12:39:51.766 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:51 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.2.service: Deactivated successfully. 2024-05-06T12:39:51.767 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:51 smithi067 systemd[1]: Stopped Ceph osd.2 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:39:51.767 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:51 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.2.service: Consumed 6.927s CPU time. 2024-05-06T12:39:51.767 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:51 smithi067 systemd[1]: Starting Ceph osd.2 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:39:52.055 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:51 smithi067 podman[72737]: 2024-05-06 12:39:51.944088131 +0000 UTC m=+0.110953529 container create 36d7b541476d932d27b8f573d5026365bbdeb11439b4b1673cfd4fd3bb787739 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:39:52.055 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:51 smithi067 podman[72737]: 2024-05-06 12:39:51.851944288 +0000 UTC m=+0.018809659 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:39:52.055 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:51 smithi067 ceph-mon[62070]: pgmap v91: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:52.055 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:51 smithi067 ceph-mon[62070]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-05-06T12:39:52.056 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:51 smithi067 ceph-mon[64635]: pgmap v91: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:52.056 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:51 smithi067 ceph-mon[64635]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-05-06T12:39:52.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:51 smithi086 ceph-mon[55252]: pgmap v91: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:39:52.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:51 smithi086 ceph-mon[55252]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-05-06T12:39:52.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 podman[72737]: 2024-05-06 12:39:52.15466204 +0000 UTC m=+0.321527387 container init 36d7b541476d932d27b8f573d5026365bbdeb11439b4b1673cfd4fd3bb787739 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-05-06T12:39:52.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 podman[72737]: 2024-05-06 12:39:52.157842252 +0000 UTC m=+0.324707593 container start 36d7b541476d932d27b8f573d5026365bbdeb11439b4b1673cfd4fd3bb787739 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430) 2024-05-06T12:39:52.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 podman[72737]: 2024-05-06 12:39:52.229646239 +0000 UTC m=+0.396511591 container attach 36d7b541476d932d27b8f573d5026365bbdeb11439b4b1673cfd4fd3bb787739 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-05-06T12:39:52.948 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate[72752]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate[72752]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 bash[72737]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 bash[72737]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 bash[72737]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 bash[72737]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 bash[72737]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 bash[72737]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 bash[72737]: --> ceph-volume raw activate successful for osd ID: 2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate[72752]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate[72752]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T12:39:52.949 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate[72752]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-05-06T12:39:52.950 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate[72752]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T12:39:52.950 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate[72752]: --> ceph-volume raw activate successful for osd ID: 2 2024-05-06T12:39:52.950 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:52 smithi067 podman[72737]: 2024-05-06 12:39:52.642520823 +0000 UTC m=+0.809386165 container died 36d7b541476d932d27b8f573d5026365bbdeb11439b4b1673cfd4fd3bb787739 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:39:52.950 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:52 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded (PG_DEGRADED) 2024-05-06T12:39:52.950 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:52 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded (PG_DEGRADED) 2024-05-06T12:39:53.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:52 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded (PG_DEGRADED) 2024-05-06T12:39:53.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 podman[72879]: 2024-05-06 12:39:53.208854548 +0000 UTC m=+0.561043989 container remove 36d7b541476d932d27b8f573d5026365bbdeb11439b4b1673cfd4fd3bb787739 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-70ce953, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:39:53.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 podman[72945]: 2024-05-06 12:39:53.329282423 +0000 UTC m=+0.018462653 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:39:53.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 podman[72945]: 2024-05-06 12:39:53.435074191 +0000 UTC m=+0.124254395 container create b6fce96724515db7ad7bf7888f1bdc251ba3165be818f39cb3c20da22c25ff2d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-06T12:39:53.823 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:39:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:39:53] "GET /metrics HTTP/1.1" 200 38481 "" "Prometheus/2.33.4" 2024-05-06T12:39:53.823 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 podman[72945]: 2024-05-06 12:39:53.637468326 +0000 UTC m=+0.326648532 container init b6fce96724515db7ad7bf7888f1bdc251ba3165be818f39cb3c20da22c25ff2d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-06T12:39:53.824 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 podman[72945]: 2024-05-06 12:39:53.640680724 +0000 UTC m=+0.329860930 container start b6fce96724515db7ad7bf7888f1bdc251ba3165be818f39cb3c20da22c25ff2d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-06T12:39:53.824 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 ceph-osd[72964]: -- 172.21.15.67:0/353792031 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56553c2c5680 con 0x56553c28f400 2024-05-06T12:39:53.824 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 bash[72945]: b6fce96724515db7ad7bf7888f1bdc251ba3165be818f39cb3c20da22c25ff2d 2024-05-06T12:39:53.824 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:53 smithi067 systemd[1]: Started Ceph osd.2 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:39:54.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:53 smithi086 ceph-mon[55252]: pgmap v92: 161 pgs: 1 active+undersized+degraded, 33 peering, 2 stale+active+clean, 6 active+undersized, 119 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5/627 objects degraded (0.797%) 2024-05-06T12:39:54.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:54.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:54.184 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:53 smithi067 ceph-mon[64635]: pgmap v92: 161 pgs: 1 active+undersized+degraded, 33 peering, 2 stale+active+clean, 6 active+undersized, 119 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5/627 objects degraded (0.797%) 2024-05-06T12:39:54.184 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:54.184 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:54.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:53 smithi067 ceph-mon[62070]: pgmap v92: 161 pgs: 1 active+undersized+degraded, 33 peering, 2 stale+active+clean, 6 active+undersized, 119 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 5/627 objects degraded (0.797%) 2024-05-06T12:39:54.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:54.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:54.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:54 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[72960]: 2024-05-06T12:39:54.182+0000 7fe7a22037c0 -1 Falling back to public interface 2024-05-06T12:39:54.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:54 smithi067 ceph-mon[62070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-05-06T12:39:54.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:54 smithi067 ceph-mon[64635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-05-06T12:39:55.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:54 smithi086 ceph-mon[55252]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-05-06T12:39:56.172 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:55 smithi067 ceph-mon[64635]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-05-06T12:39:56.172 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:55 smithi067 ceph-mon[62070]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-05-06T12:39:56.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:55 smithi086 ceph-mon[55252]: pgmap v93: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-05-06T12:39:57.182 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[72960]: 2024-05-06T12:39:56.899+0000 7fe7a22037c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-05-06T12:39:57.183 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[72960]: 2024-05-06T12:39:56.926+0000 7fe7a22037c0 -1 osd.2 115 log_to_monitors true 2024-05-06T12:39:57.915 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:57 smithi067 ceph-mon[62070]: pgmap v94: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-05-06T12:39:57.915 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:57 smithi067 ceph-mon[62070]: from='osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:39:57.915 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:57 smithi067 ceph-mon[62070]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:39:57.916 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:57 smithi067 ceph-mon[64635]: pgmap v94: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-05-06T12:39:57.916 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:57 smithi067 ceph-mon[64635]: from='osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:39:57.916 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:57 smithi067 ceph-mon[64635]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:39:58.225 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:39:57 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[72960]: 2024-05-06T12:39:57.913+0000 7fe79365a700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:39:58.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:57 smithi086 ceph-mon[55252]: pgmap v94: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-05-06T12:39:58.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:57 smithi086 ceph-mon[55252]: from='osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:39:58.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:57 smithi086 ceph-mon[55252]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T12:39:59.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:39:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:39:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:39:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:58 smithi086 ceph-mon[55252]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T12:39:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:58 smithi086 ceph-mon[55252]: osdmap e118: 8 total, 7 up, 8 in 2024-05-06T12:39:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:58 smithi086 ceph-mon[55252]: from='osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:58 smithi086 ceph-mon[55252]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:58 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:58 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:59.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:58 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[62070]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[62070]: osdmap e118: 8 total, 7 up, 8 in 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[62070]: from='osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[62070]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T12:39:59.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[64635]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T12:39:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[64635]: osdmap e118: 8 total, 7 up, 8 in 2024-05-06T12:39:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[64635]: from='osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[64635]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:39:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:39:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:58 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: pgmap v96: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807] boot 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: osdmap e119: 8 total, 8 up, 8 in 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[62070]: osdmap e120: 8 total, 8 up, 8 in 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: pgmap v96: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807] boot 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: osdmap e119: 8 total, 8 up, 8 in 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:00.044 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:39:59 smithi067 ceph-mon[64635]: osdmap e120: 8 total, 8 up, 8 in 2024-05-06T12:40:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: pgmap v96: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-05-06T12:40:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:40:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: osd.2 [v2:172.21.15.67:6818/2532753807,v1:172.21.15.67:6819/2532753807] boot 2024-05-06T12:40:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: osdmap e119: 8 total, 8 up, 8 in 2024-05-06T12:40:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T12:40:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:39:59 smithi086 ceph-mon[55252]: osdmap e120: 8 total, 8 up, 8 in 2024-05-06T12:40:01.161 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:01.161 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: Health detail: HEALTH_WARN Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded 2024-05-06T12:40:01.161 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: [WRN] PG_DEGRADED: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded 2024-05-06T12:40:01.161 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-05-06T12:40:01.161 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 2.f is active+undersized+degraded, acting [4,0] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 4.d is active+undersized+degraded, acting [4,1] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 5.d is active+undersized+degraded, acting [7,5] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[64635]: pg 5.1c is active+undersized+degraded, acting [4,3] 2024-05-06T12:40:01.162 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: Health detail: HEALTH_WARN Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: [WRN] PG_DEGRADED: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 2.f is active+undersized+degraded, acting [4,0] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 4.d is active+undersized+degraded, acting [4,1] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 5.d is active+undersized+degraded, acting [7,5] 2024-05-06T12:40:01.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:00 smithi067 ceph-mon[62070]: pg 5.1c is active+undersized+degraded, acting [4,3] 2024-05-06T12:40:01.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: Health detail: HEALTH_WARN Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: [WRN] PG_DEGRADED: Degraded data redundancy: 55/627 objects degraded (8.772%), 13 pgs degraded 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 2.f is active+undersized+degraded, acting [4,0] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-05-06T12:40:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-05-06T12:40:01.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 4.d is active+undersized+degraded, acting [4,1] 2024-05-06T12:40:01.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 5.d is active+undersized+degraded, acting [7,5] 2024-05-06T12:40:01.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:00 smithi086 ceph-mon[55252]: pg 5.1c is active+undersized+degraded, acting [4,3] 2024-05-06T12:40:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:01 smithi067 ceph-mon[64635]: pgmap v99: 161 pgs: 25 peering, 5 active+undersized+degraded, 17 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-06T12:40:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:01 smithi067 ceph-mon[62070]: pgmap v99: 161 pgs: 25 peering, 5 active+undersized+degraded, 17 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-06T12:40:02.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:01 smithi086 ceph-mon[55252]: pgmap v99: 161 pgs: 25 peering, 5 active+undersized+degraded, 17 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-06T12:40:02.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:02 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:40:02.815+0000 7f337218f700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:40:03.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:40:03] "GET /metrics HTTP/1.1" 200 38481 "" "Prometheus/2.33.4" 2024-05-06T12:40:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:03.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: pgmap v100: 161 pgs: 25 peering, 4 active+undersized+degraded, 13 active+undersized, 119 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 21/627 objects degraded (3.349%) 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:03.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: pgmap v100: 161 pgs: 25 peering, 4 active+undersized+degraded, 13 active+undersized, 119 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 21/627 objects degraded (3.349%) 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:40:03.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:03 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: pgmap v100: 161 pgs: 25 peering, 4 active+undersized+degraded, 13 active+undersized, 119 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 21/627 objects degraded (3.349%) 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T12:40:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:03 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 26/627 objects degraded (4.147%), 5 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:05.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:05.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:05.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:05.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:05.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:05.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:05.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:05.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:05.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:05.319 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:40:05.998 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:40:06.101 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (4m) 7s ago 9m 24.9M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:40:06.101 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (3m) 2m ago 8m 43.8M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:40:06.101 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (3m) 7s ago 8m 52.9M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:40:06.101 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (3m) 2m ago 13m 449M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:40:06.101 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (3m) 7s ago 15m 507M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (2m) 7s ago 15m 47.8M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (2m) 2m ago 13m 19.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (2m) 7s ago 14m 39.8M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (4m) 7s ago 9m 17.3M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (3m) 2m ago 9m 16.0M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (83s) 7s ago 13m 51.2M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (47s) 7s ago 13m 50.7M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (12s) 7s ago 12m 12.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (12m) 7s ago 12m 55.0M 2736M 17.2.0 e1d6a67b021e 75783ba6cea6 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (11m) 2m ago 11m 53.3M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (11m) 2m ago 11m 50.3M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (11m) 2m ago 11m 51.2M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (10m) 2m ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:40:06.102 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (3m) 2m ago 10m 58.2M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:40:06.103 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (8m) 7s ago 8m 75.4M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:40:06.103 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (8m) 2m ago 8m 72.4M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:40:06.103 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (8m) 7s ago 8m 74.9M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:40:06.103 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (8m) 2m ago 8m 72.5M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:40:06.103 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[62070]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T12:40:06.103 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/627 objects degraded (3.349%), 4 pgs degraded) 2024-05-06T12:40:06.103 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:40:06.104 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[62070]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:06.104 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[64635]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T12:40:06.104 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/627 objects degraded (3.349%), 4 pgs degraded) 2024-05-06T12:40:06.104 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:40:06.104 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:05 smithi067 ceph-mon[64635]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:05 smithi086 ceph-mon[55252]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T12:40:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:05 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/627 objects degraded (3.349%), 4 pgs degraded) 2024-05-06T12:40:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:05 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:40:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:05 smithi086 ceph-mon[55252]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:06.412 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:40:06.413 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:40:06.759 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:40:06.759 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:40:06.759 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:40:06.759 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:40:06.759 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:40:06.759 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:40:06.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:40:06.760 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:40:06.760 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "8/23 daemons upgraded", 2024-05-06T12:40:06.760 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:40:06.760 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:40:06.760 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:40:07.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:06 smithi086 ceph-mon[55252]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:07.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:06 smithi086 ceph-mon[55252]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:07.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:06 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/2395696474' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:07.169 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:40:07.169 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:06 smithi067 ceph-mon[64635]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:07.169 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:06 smithi067 ceph-mon[64635]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:07.169 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:06 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/2395696474' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:07.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:06 smithi067 ceph-mon[62070]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:07.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:06 smithi067 ceph-mon[62070]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:07.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:06 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/2395696474' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:08.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:07 smithi086 ceph-mon[55252]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:08.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:07 smithi086 ceph-mon[55252]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:08.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:07 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/462091661' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:40:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:07 smithi067 ceph-mon[62070]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:07 smithi067 ceph-mon[62070]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:07 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/462091661' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:40:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:07 smithi067 ceph-mon[64635]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:07 smithi067 ceph-mon[64635]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:07 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/462091661' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:40:09.021 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:40:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:40:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:40:10.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:09 smithi086 ceph-mon[55252]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:40:10.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:09 smithi067 ceph-mon[62070]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:40:10.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:09 smithi067 ceph-mon[64635]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:40:11.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:10 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:10 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:11.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:10 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:12.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:11 smithi086 ceph-mon[55252]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-05-06T12:40:12.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:11 smithi067 ceph-mon[62070]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-05-06T12:40:12.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:11 smithi067 ceph-mon[64635]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-05-06T12:40:14.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:13 smithi086 ceph-mon[55252]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:14.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:13 smithi067 ceph-mon[62070]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:14.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:40:13] "GET /metrics HTTP/1.1" 200 38492 "" "Prometheus/2.33.4" 2024-05-06T12:40:14.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:13 smithi067 ceph-mon[64635]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:15.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:15 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:15.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:15 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:15.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:15 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:16.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:16 smithi086 ceph-mon[55252]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:16.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:16 smithi067 ceph-mon[62070]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:16 smithi067 ceph-mon[64635]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:18.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:18 smithi086 ceph-mon[55252]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:18.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:18.441 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:18 smithi067 ceph-mon[64635]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:18.441 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:18.441 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:18 smithi067 ceph-mon[62070]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:18.441 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:19.029 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:40:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:40:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:40:19.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:19.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[62070]: Upgrade: osd.3 is safe to restart 2024-05-06T12:40:19.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:19.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:19.322 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:19.322 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[64635]: Upgrade: osd.3 is safe to restart 2024-05-06T12:40:19.322 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:19.322 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:19.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:19 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T12:40:19.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:19 smithi086 ceph-mon[55252]: Upgrade: osd.3 is safe to restart 2024-05-06T12:40:19.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:19.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:20.687 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[64635]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:20.687 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[64635]: Upgrade: Updating osd.3 2024-05-06T12:40:20.687 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:20.687 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:40:20.687 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:20.687 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[64635]: Deploying daemon osd.3 on smithi067 2024-05-06T12:40:20.687 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[62070]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:20.688 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[62070]: Upgrade: Updating osd.3 2024-05-06T12:40:20.688 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:20.688 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:40:20.688 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:20.688 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:20 smithi067 ceph-mon[62070]: Deploying daemon osd.3 on smithi067 2024-05-06T12:40:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:20 smithi086 ceph-mon[55252]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:20 smithi086 ceph-mon[55252]: Upgrade: Updating osd.3 2024-05-06T12:40:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T12:40:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:20 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:20 smithi086 ceph-mon[55252]: Deploying daemon osd.3 on smithi067 2024-05-06T12:40:21.649 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:21 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:21.650 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:21 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:21.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:21 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:22 smithi067 ceph-mon[62070]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:22.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:22 smithi067 ceph-mon[64635]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:22.726 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:22 smithi067 systemd[1]: Stopping Ceph osd.3 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:40:22.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:22 smithi086 ceph-mon[55252]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:23.225 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[38867]: 2024-05-06T12:40:22.824+0000 7fc39802b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:40:23.225 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[38867]: 2024-05-06T12:40:22.824+0000 7fc39802b700 -1 osd.3 120 *** Got signal Terminated *** 2024-05-06T12:40:23.225 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[38867]: 2024-05-06T12:40:22.824+0000 7fc39802b700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:40:23.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:23 smithi067 ceph-mon[62070]: osd.3 marked itself down and dead 2024-05-06T12:40:23.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:23 smithi067 ceph-mon[64635]: osd.3 marked itself down and dead 2024-05-06T12:40:23.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:23 smithi086 ceph-mon[55252]: osd.3 marked itself down and dead 2024-05-06T12:40:24.167 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:40:23] "GET /metrics HTTP/1.1" 200 38490 "" "Prometheus/2.33.4" 2024-05-06T12:40:24.167 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:23 smithi067 podman[74961]: 2024-05-06 12:40:23.892593739 +0000 UTC m=+1.166812990 container died 75783ba6cea69a7b9bbe78434b648a8b2189b586da16accebc1acaa6b600921d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-05-06T12:40:24.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:24 smithi067 ceph-mon[62070]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:24.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:24 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:40:24.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:24 smithi067 ceph-mon[62070]: osdmap e121: 8 total, 7 up, 8 in 2024-05-06T12:40:24.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:24 smithi067 ceph-mon[64635]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:24.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:24 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:40:24.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:24 smithi067 ceph-mon[64635]: osdmap e121: 8 total, 7 up, 8 in 2024-05-06T12:40:24.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:24 smithi067 podman[74961]: 2024-05-06 12:40:24.265336322 +0000 UTC m=+1.539555559 container cleanup 75783ba6cea69a7b9bbe78434b648a8b2189b586da16accebc1acaa6b600921d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, release=754, ceph=True, build-date=2022-05-03T08:36:31.336870) 2024-05-06T12:40:24.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:24 smithi067 bash[74961]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3 2024-05-06T12:40:24.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:24 smithi086 ceph-mon[55252]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:24.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:24 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:40:24.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:24 smithi086 ceph-mon[55252]: osdmap e121: 8 total, 7 up, 8 in 2024-05-06T12:40:24.882 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:24 smithi067 podman[74989]: 2024-05-06 12:40:24.55897156 +0000 UTC m=+0.662037107 container remove 75783ba6cea69a7b9bbe78434b648a8b2189b586da16accebc1acaa6b600921d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, ceph=True, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8) 2024-05-06T12:40:25.205 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:24 smithi067 podman[75083]: 2024-05-06 12:40:24.785251405 +0000 UTC m=+0.019168017 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:40:25.205 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:24 smithi067 podman[75083]: 2024-05-06 12:40:24.893332531 +0000 UTC m=+0.127249196 container create ec5a920e09c7a1398b808fffd4ab5c038184a2ba2d9f141c16fab1523694a7f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:40:25.205 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:25 smithi067 podman[75083]: 2024-05-06 12:40:25.121082118 +0000 UTC m=+0.354998737 container init ec5a920e09c7a1398b808fffd4ab5c038184a2ba2d9f141c16fab1523694a7f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240430) 2024-05-06T12:40:25.205 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:25 smithi067 podman[75083]: 2024-05-06 12:40:25.124665873 +0000 UTC m=+0.358582478 container start ec5a920e09c7a1398b808fffd4ab5c038184a2ba2d9f141c16fab1523694a7f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:40:25.205 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:25 smithi067 podman[75083]: 2024-05-06 12:40:25.204016941 +0000 UTC m=+0.437933555 container attach ec5a920e09c7a1398b808fffd4ab5c038184a2ba2d9f141c16fab1523694a7f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:40:25.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:25 smithi067 ceph-mon[62070]: osdmap e122: 8 total, 7 up, 8 in 2024-05-06T12:40:25.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:25 smithi067 ceph-mon[64635]: osdmap e122: 8 total, 7 up, 8 in 2024-05-06T12:40:25.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:25 smithi067 podman[75083]: 2024-05-06 12:40:25.432453165 +0000 UTC m=+0.666369768 container died ec5a920e09c7a1398b808fffd4ab5c038184a2ba2d9f141c16fab1523694a7f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-06T12:40:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:25 smithi086 ceph-mon[55252]: osdmap e122: 8 total, 7 up, 8 in 2024-05-06T12:40:26.428 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:25 smithi067 podman[75109]: 2024-05-06 12:40:25.999620498 +0000 UTC m=+0.559334878 container remove ec5a920e09c7a1398b808fffd4ab5c038184a2ba2d9f141c16fab1523694a7f3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-06T12:40:26.428 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.3.service: Deactivated successfully. 2024-05-06T12:40:26.428 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 systemd[1]: Stopped Ceph osd.3 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:40:26.429 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.3.service: Consumed 8.590s CPU time. 2024-05-06T12:40:26.429 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 systemd[1]: Starting Ceph osd.3 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:40:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:26 smithi067 ceph-mon[62070]: pgmap v113: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 19 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-05-06T12:40:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:26 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:26 smithi067 ceph-mon[64635]: pgmap v113: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 19 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-05-06T12:40:26.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:26 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:26.726 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 podman[75212]: 2024-05-06 12:40:26.364690212 +0000 UTC m=+0.018684265 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:40:26.726 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 podman[75212]: 2024-05-06 12:40:26.466422711 +0000 UTC m=+0.120416748 container create d3eb8b6991588e1e537a4bab949898256f59922fcf07db7f705cfa49d823c167 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-06T12:40:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:26 smithi086 ceph-mon[55252]: pgmap v113: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 19 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-05-06T12:40:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:26 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:27.195 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 podman[75212]: 2024-05-06 12:40:26.732905048 +0000 UTC m=+0.386899087 container init d3eb8b6991588e1e537a4bab949898256f59922fcf07db7f705cfa49d823c167 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-06T12:40:27.195 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 podman[75212]: 2024-05-06 12:40:26.736802759 +0000 UTC m=+0.390796800 container start d3eb8b6991588e1e537a4bab949898256f59922fcf07db7f705cfa49d823c167 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:40:27.195 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:26 smithi067 podman[75212]: 2024-05-06 12:40:26.820375591 +0000 UTC m=+0.474369640 container attach d3eb8b6991588e1e537a4bab949898256f59922fcf07db7f705cfa49d823c167 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-06T12:40:27.475 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate[75227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T12:40:27.475 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate[75227]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T12:40:27.475 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 bash[75212]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 bash[75212]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 bash[75212]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 bash[75212]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 bash[75212]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 bash[75212]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 bash[75212]: --> ceph-volume raw activate successful for osd ID: 3 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate[75227]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate[75227]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate[75227]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate[75227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate[75227]: --> ceph-volume raw activate successful for osd ID: 3 2024-05-06T12:40:27.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 podman[75212]: 2024-05-06 12:40:27.218998188 +0000 UTC m=+0.872992229 container died d3eb8b6991588e1e537a4bab949898256f59922fcf07db7f705cfa49d823c167 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-06T12:40:28.186 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:27 smithi067 podman[75348]: 2024-05-06 12:40:27.851459075 +0000 UTC m=+0.626589052 container remove d3eb8b6991588e1e537a4bab949898256f59922fcf07db7f705cfa49d823c167 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-06T12:40:28.186 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 podman[75415]: 2024-05-06 12:40:27.977739364 +0000 UTC m=+0.019348130 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:40:28.186 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 podman[75415]: 2024-05-06 12:40:28.077444197 +0000 UTC m=+0.119053000 container create 7ec6d2a3db7332ab32a4df1b6b40889bc209832ef1bdb7a9f8eaca25d10d8e79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:40:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:28 smithi067 ceph-mon[62070]: pgmap v114: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 19 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-05-06T12:40:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:28 smithi067 ceph-mon[64635]: pgmap v114: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 19 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-05-06T12:40:28.475 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 podman[75415]: 2024-05-06 12:40:28.312881222 +0000 UTC m=+0.354489986 container init 7ec6d2a3db7332ab32a4df1b6b40889bc209832ef1bdb7a9f8eaca25d10d8e79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-06T12:40:28.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 podman[75415]: 2024-05-06 12:40:28.316126622 +0000 UTC m=+0.357735384 container start 7ec6d2a3db7332ab32a4df1b6b40889bc209832ef1bdb7a9f8eaca25d10d8e79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-06T12:40:28.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 ceph-osd[75434]: -- 172.21.15.67:0/3913609309 <== mon.1 v2:172.21.15.67:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561764489680 con 0x561764453000 2024-05-06T12:40:28.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 bash[75415]: 7ec6d2a3db7332ab32a4df1b6b40889bc209832ef1bdb7a9f8eaca25d10d8e79 2024-05-06T12:40:28.476 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 systemd[1]: Started Ceph osd.3 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:40:28.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:28 smithi086 ceph-mon[55252]: pgmap v114: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 19 active+undersized, 116 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-05-06T12:40:28.869 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:40:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:40:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:40:29.225 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:28 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[75430]: 2024-05-06T12:40:28.858+0000 7fd2b55187c0 -1 Falling back to public interface 2024-05-06T12:40:29.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:29 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:29.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:29 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:29.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:29 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:29.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:29 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:29.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:29 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:29.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:29 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:30.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:30 smithi067 ceph-mon[64635]: pgmap v115: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:30.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:30 smithi067 ceph-mon[62070]: pgmap v115: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:30 smithi086 ceph-mon[55252]: pgmap v115: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:31.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:31 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:31.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:31 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:31.725 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:31 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[75430]: 2024-05-06T12:40:31.570+0000 7fd2b55187c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-05-06T12:40:31.725 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:31 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[75430]: 2024-05-06T12:40:31.598+0000 7fd2b55187c0 -1 osd.3 120 log_to_monitors true 2024-05-06T12:40:31.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:31 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:32.725 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:40:32 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[75430]: 2024-05-06T12:40:32.504+0000 7fd2a696f700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:40:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:32 smithi067 ceph-mon[62070]: pgmap v116: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:32 smithi067 ceph-mon[62070]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:40:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:32 smithi067 ceph-mon[64635]: pgmap v116: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:32.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:32 smithi067 ceph-mon[64635]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:40:32.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:32 smithi086 ceph-mon[55252]: pgmap v116: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:32.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:32 smithi086 ceph-mon[55252]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T12:40:33.745 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[64635]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T12:40:33.745 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[64635]: osdmap e123: 8 total, 7 up, 8 in 2024-05-06T12:40:33.745 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[64635]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:40:33.745 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:33.745 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:33.745 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 25 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:33.745 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[62070]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T12:40:33.746 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[62070]: osdmap e123: 8 total, 7 up, 8 in 2024-05-06T12:40:33.746 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[62070]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:40:33.746 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:33.746 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:33.746 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:33 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 25 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:33.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:33 smithi086 ceph-mon[55252]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T12:40:33.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:33 smithi086 ceph-mon[55252]: osdmap e123: 8 total, 7 up, 8 in 2024-05-06T12:40:33.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:33 smithi086 ceph-mon[55252]: from='osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-06T12:40:33.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:33.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:33.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:33 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 25 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:34.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:40:33] "GET /metrics HTTP/1.1" 200 38490 "" "Prometheus/2.33.4" 2024-05-06T12:40:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: pgmap v118: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:40:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350] boot 2024-05-06T12:40:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: osdmap e124: 8 total, 8 up, 8 in 2024-05-06T12:40:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:40:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: pgmap v118: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350] boot 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: osdmap e124: 8 total, 8 up, 8 in 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: pgmap v118: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: osd.3 [v2:172.21.15.67:6826/3351723350,v1:172.21.15.67:6827/3351723350] boot 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: osdmap e124: 8 total, 8 up, 8 in 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:35.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:35 smithi086 ceph-mon[55252]: osdmap e125: 8 total, 8 up, 8 in 2024-05-06T12:40:35.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:35 smithi067 ceph-mon[62070]: osdmap e125: 8 total, 8 up, 8 in 2024-05-06T12:40:35.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:35 smithi067 ceph-mon[64635]: osdmap e125: 8 total, 8 up, 8 in 2024-05-06T12:40:36.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:36 smithi067 ceph-mon[62070]: pgmap v121: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 62/627 objects degraded (9.888%) 2024-05-06T12:40:36.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:36 smithi067 ceph-mon[64635]: pgmap v121: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 62/627 objects degraded (9.888%) 2024-05-06T12:40:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:36 smithi086 ceph-mon[55252]: pgmap v121: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 62/627 objects degraded (9.888%) 2024-05-06T12:40:37.540 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:40:37.781 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:37 smithi067 ceph-mon[62070]: pgmap v122: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-05-06T12:40:37.781 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:37 smithi067 ceph-mon[64635]: pgmap v122: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-05-06T12:40:37.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:37 smithi086 ceph-mon[55252]: pgmap v122: 161 pgs: 17 active+undersized+degraded, 31 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-05-06T12:40:38.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:38 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:40:38.016+0000 7f337218f700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-05-06T12:40:38.248 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:40:38.248 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (4m) 5s ago 9m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:40:38.248 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (4m) 2m ago 9m 43.8M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:40:38.248 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (3m) 5s ago 8m 53.1M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (3m) 2m ago 14m 449M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (4m) 5s ago 15m 508M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (3m) 5s ago 16m 47.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (2m) 2m ago 14m 19.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (3m) 5s ago 14m 38.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (4m) 5s ago 10m 18.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (4m) 2m ago 10m 16.0M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:40:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (115s) 5s ago 13m 51.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:40:38.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (80s) 5s ago 13m 52.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:40:38.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (44s) 5s ago 13m 40.8M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:40:38.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (10s) 5s ago 12m 14.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:40:38.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (12m) 2m ago 12m 53.3M 2992M 17.2.0 e1d6a67b021e f9f80e97c997 2024-05-06T12:40:38.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (12m) 2m ago 12m 50.3M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:40:38.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (11m) 2m ago 11m 51.2M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:40:38.251 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (11m) 2m ago 11m 54.5M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:40:38.251 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (3m) 2m ago 10m 58.2M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:40:38.251 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (8m) 5s ago 8m 75.5M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:40:38.251 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (8m) 2m ago 8m 72.4M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:40:38.251 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (8m) 5s ago 8m 75.1M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:40:38.251 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (8m) 2m ago 8m 72.5M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:40:38.659 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:40:38.659 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:40:38.660 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:40:38.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:38.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:40:38.660 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:40:38.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:38.660 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:40:38.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-05-06T12:40:38.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 4 2024-05-06T12:40:38.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:38.661 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:40:38.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:40:38.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:40:38.661 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:40:38.662 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-05-06T12:40:38.662 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 9 2024-05-06T12:40:38.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:40:38.662 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:40:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:40:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 62/627 objects degraded (9.888%), 17 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:38.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3387745198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:38.977 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-05-06T12:40:38.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:38.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 62/627 objects degraded (9.888%), 17 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:38.978 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:38 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3387745198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "9/23 daemons upgraded", 2024-05-06T12:40:39.009 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:40:39.010 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:40:39.010 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:40:39.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:40:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:40:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:40:39.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-05-06T12:40:39.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:39.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 62/627 objects degraded (9.888%), 17 pgs degraded (PG_DEGRADED) 2024-05-06T12:40:39.121 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:38 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3387745198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:40:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:40:40.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:39 smithi086 ceph-mon[55252]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:40:40.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:39 smithi086 ceph-mon[55252]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:40.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:39 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/627 objects degraded (9.888%), 17 pgs degraded) 2024-05-06T12:40:40.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:39 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:40:40.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:39 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1441299930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:40:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[62070]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:40:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[62070]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/627 objects degraded (9.888%), 17 pgs degraded) 2024-05-06T12:40:40.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:40:40.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1441299930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:40:40.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[64635]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:40:40.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[64635]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:40:40.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/627 objects degraded (9.888%), 17 pgs degraded) 2024-05-06T12:40:40.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:40:40.227 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:39 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1441299930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:40:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:40 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:40 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:41.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:40 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:41 smithi067 ceph-mon[62070]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:40:42.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:41 smithi067 ceph-mon[64635]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:40:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:41 smithi086 ceph-mon[55252]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-06T12:40:44.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:40:43] "GET /metrics HTTP/1.1" 200 38360 "" "Prometheus/2.33.4" 2024-05-06T12:40:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:43 smithi067 ceph-mon[64635]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-05-06T12:40:44.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:43 smithi067 ceph-mon[62070]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-05-06T12:40:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:43 smithi086 ceph-mon[55252]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 222 B/s rd, 0 op/s 2024-05-06T12:40:45.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:45 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:45.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:45 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:45.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:45 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:46 smithi086 ceph-mon[55252]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-05-06T12:40:46.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:46 smithi067 ceph-mon[64635]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-05-06T12:40:46.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:46 smithi067 ceph-mon[62070]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2024-05-06T12:40:48.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:48 smithi086 ceph-mon[55252]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T12:40:48.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:48 smithi067 ceph-mon[62070]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T12:40:48.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:48 smithi067 ceph-mon[64635]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T12:40:49.119 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:40:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:40:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:40:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:49 smithi086 ceph-mon[55252]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T12:40:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:50.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:49 smithi067 ceph-mon[62070]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T12:40:50.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:50.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:50.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:49 smithi067 ceph-mon[64635]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T12:40:50.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:50.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:40:51.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:50 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:51.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:50 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:51.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:50 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:40:52.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:51 smithi086 ceph-mon[55252]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:52.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:51 smithi067 ceph-mon[62070]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:52.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:51 smithi067 ceph-mon[64635]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:53.909 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:53 smithi086 ceph-mon[55252]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:53.909 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:53.909 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:53 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:53.909 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:53 smithi086 ceph-mon[55252]: Upgrade: osd.4 is safe to restart 2024-05-06T12:40:54.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:40:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:40:53] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.33.4" 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[62070]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[62070]: Upgrade: osd.4 is safe to restart 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[64635]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T12:40:54.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:53 smithi067 ceph-mon[64635]: Upgrade: osd.4 is safe to restart 2024-05-06T12:40:55.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:55 smithi086 ceph-mon[55252]: Upgrade: Updating osd.4 2024-05-06T12:40:55.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:55.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:40:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:55 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:55 smithi086 ceph-mon[55252]: Deploying daemon osd.4 on smithi086 2024-05-06T12:40:55.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:55 smithi086 ceph-mon[55252]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:55.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[62070]: Upgrade: Updating osd.4 2024-05-06T12:40:55.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:55.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:40:55.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:55.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[62070]: Deploying daemon osd.4 on smithi086 2024-05-06T12:40:55.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[62070]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:55.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[64635]: Upgrade: Updating osd.4 2024-05-06T12:40:55.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:40:55.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T12:40:55.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:40:55.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[64635]: Deploying daemon osd.4 on smithi086 2024-05-06T12:40:55.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:55 smithi067 ceph-mon[64635]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:58.040 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:57 smithi067 ceph-mon[62070]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:58.041 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:57 smithi067 ceph-mon[64635]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:58.060 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:57 smithi086 ceph-mon[55252]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:40:58.060 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:57 smithi086 systemd[1]: Stopping Ceph osd.4 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:40:58.369 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[26532]: 2024-05-06T12:40:58.079+0000 7f8e54708700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:40:58.369 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[26532]: 2024-05-06T12:40:58.079+0000 7f8e54708700 -1 osd.4 125 *** Got signal Terminated *** 2024-05-06T12:40:58.370 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[26532]: 2024-05-06T12:40:58.079+0000 7f8e54708700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:40:58.982 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:40:58 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:40:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"smithi067:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.67\", device_class=\"ssd\", hostname=\"smithi067\", instance=\"172.21.15.86:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.67\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T12:40:58.982 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:58 smithi086 ceph-mon[55252]: osd.4 marked itself down and dead 2024-05-06T12:40:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:58 smithi067 ceph-mon[62070]: osd.4 marked itself down and dead 2024-05-06T12:40:59.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:58 smithi067 ceph-mon[64635]: osd.4 marked itself down and dead 2024-05-06T12:40:59.369 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:59 smithi086 podman[58062]: 2024-05-06 12:40:59.142295304 +0000 UTC m=+1.165761006 container died f9f80e97c997ce9b0995373a64255c4f18f74abaec780cd7a07b537d60b4712f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, architecture=x86_64, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public) 2024-05-06T12:40:59.766 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:59 smithi086 podman[58062]: 2024-05-06 12:40:59.489033325 +0000 UTC m=+1.512499024 container cleanup f9f80e97c997ce9b0995373a64255c4f18f74abaec780cd7a07b537d60b4712f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, name=centos-stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, version=8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-06T12:40:59.766 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:59 smithi086 bash[58062]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4 2024-05-06T12:41:00.109 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:59 smithi086 ceph-mon[55252]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:00.110 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:59 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:41:00.110 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:40:59 smithi086 ceph-mon[55252]: osdmap e126: 8 total, 7 up, 8 in 2024-05-06T12:41:00.110 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:40:59 smithi086 podman[58090]: 2024-05-06 12:40:59.765369796 +0000 UTC m=+0.620684576 container remove f9f80e97c997ce9b0995373a64255c4f18f74abaec780cd7a07b537d60b4712f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, architecture=x86_64, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, release=754, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-06T12:41:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:59 smithi067 ceph-mon[62070]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:59 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:41:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:40:59 smithi067 ceph-mon[62070]: osdmap e126: 8 total, 7 up, 8 in 2024-05-06T12:41:00.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:59 smithi067 ceph-mon[64635]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:00.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:59 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:41:00.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:40:59 smithi067 ceph-mon[64635]: osdmap e126: 8 total, 7 up, 8 in 2024-05-06T12:41:00.369 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:00 smithi086 podman[58181]: 2024-05-06 12:41:00.011745987 +0000 UTC m=+0.017591075 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:41:00.369 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:00 smithi086 podman[58181]: 2024-05-06 12:41:00.141340153 +0000 UTC m=+0.147185224 container create 6790626cc429076dae4ccfca9ba1bb910bf09f03307721be473825edede8e02e (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240430, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:41:00.676 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:00 smithi086 podman[58181]: 2024-05-06 12:41:00.386363841 +0000 UTC m=+0.392208914 container init 6790626cc429076dae4ccfca9ba1bb910bf09f03307721be473825edede8e02e (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240430, ceph=True, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-06T12:41:00.677 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:00 smithi086 podman[58181]: 2024-05-06 12:41:00.389908605 +0000 UTC m=+0.395753679 container start 6790626cc429076dae4ccfca9ba1bb910bf09f03307721be473825edede8e02e (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-05-06T12:41:00.677 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:00 smithi086 podman[58181]: 2024-05-06 12:41:00.469216079 +0000 UTC m=+0.475061165 container attach 6790626cc429076dae4ccfca9ba1bb910bf09f03307721be473825edede8e02e (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-06T12:41:00.973 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:00 smithi086 podman[58181]: 2024-05-06 12:41:00.675990713 +0000 UTC m=+0.681835784 container died 6790626cc429076dae4ccfca9ba1bb910bf09f03307721be473825edede8e02e (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-06T12:41:00.974 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:00 smithi086 ceph-mon[55252]: osdmap e127: 8 total, 7 up, 8 in 2024-05-06T12:41:00.974 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:00 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:00 smithi067 ceph-mon[62070]: osdmap e127: 8 total, 7 up, 8 in 2024-05-06T12:41:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:00 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:01.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:00 smithi067 ceph-mon[64635]: osdmap e127: 8 total, 7 up, 8 in 2024-05-06T12:41:01.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:00 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:01.619 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 podman[58207]: 2024-05-06 12:41:01.247658823 +0000 UTC m=+0.566394245 container remove 6790626cc429076dae4ccfca9ba1bb910bf09f03307721be473825edede8e02e (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-06T12:41:01.619 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.4.service: Deactivated successfully. 2024-05-06T12:41:01.619 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 systemd[1]: Stopped Ceph osd.4 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:41:01.619 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.4.service: Consumed 8.680s CPU time. 2024-05-06T12:41:01.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 systemd[1]: Starting Ceph osd.4 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:41:01.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:01 smithi086 ceph-mon[55252]: pgmap v136: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T12:41:01.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:01 smithi086 ceph-mon[55252]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-06T12:41:01.969 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 podman[58311]: 2024-05-06 12:41:01.665816994 +0000 UTC m=+0.113736413 container create 2b306daa877f1028a8c02ec09d62a38e4b1bb9f008227253a7534b89116753a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-06T12:41:01.969 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 podman[58311]: 2024-05-06 12:41:01.569376984 +0000 UTC m=+0.017296377 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:41:01.969 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 podman[58311]: 2024-05-06 12:41:01.885548506 +0000 UTC m=+0.333467884 container init 2b306daa877f1028a8c02ec09d62a38e4b1bb9f008227253a7534b89116753a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, ceph=True, io.buildah.version=1.33.5) 2024-05-06T12:41:01.969 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 podman[58311]: 2024-05-06 12:41:01.888952069 +0000 UTC m=+0.336871454 container start 2b306daa877f1028a8c02ec09d62a38e4b1bb9f008227253a7534b89116753a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-06T12:41:01.969 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:01 smithi086 podman[58311]: 2024-05-06 12:41:01.968294094 +0000 UTC m=+0.416213482 container attach 2b306daa877f1028a8c02ec09d62a38e4b1bb9f008227253a7534b89116753a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-06T12:41:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:01 smithi067 ceph-mon[64635]: pgmap v136: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T12:41:02.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:01 smithi067 ceph-mon[64635]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-06T12:41:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:01 smithi067 ceph-mon[62070]: pgmap v136: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T12:41:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:01 smithi067 ceph-mon[62070]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-06T12:41:02.619 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate[58326]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate[58326]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 bash[58311]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 bash[58311]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 bash[58311]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 bash[58311]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 bash[58311]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 bash[58311]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 bash[58311]: --> ceph-volume raw activate successful for osd ID: 4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate[58326]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate[58326]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate[58326]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate[58326]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T12:41:02.620 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate[58326]: --> ceph-volume raw activate successful for osd ID: 4 2024-05-06T12:41:02.621 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 podman[58311]: 2024-05-06 12:41:02.352962175 +0000 UTC m=+0.800881558 container died 2b306daa877f1028a8c02ec09d62a38e4b1bb9f008227253a7534b89116753a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953) 2024-05-06T12:41:02.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:02 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 29/627 objects degraded (4.625%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:03.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:02 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 29/627 objects degraded (4.625%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:03.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:02 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 29/627 objects degraded (4.625%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:03.334 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:02 smithi086 podman[58449]: 2024-05-06 12:41:02.989089777 +0000 UTC m=+0.631235320 container remove 2b306daa877f1028a8c02ec09d62a38e4b1bb9f008227253a7534b89116753a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430) 2024-05-06T12:41:03.335 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 podman[58513]: 2024-05-06 12:41:03.206598403 +0000 UTC m=+0.115101450 container create 2e371a2f9616261e0dd6006914f213ed9297c204627189457448fc67fb26f821 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-06T12:41:03.335 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 podman[58513]: 2024-05-06 12:41:03.109487325 +0000 UTC m=+0.017990344 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:41:03.840 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 podman[58513]: 2024-05-06 12:41:03.41740069 +0000 UTC m=+0.325903695 container init 2e371a2f9616261e0dd6006914f213ed9297c204627189457448fc67fb26f821 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240430) 2024-05-06T12:41:03.840 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 podman[58513]: 2024-05-06 12:41:03.420362523 +0000 UTC m=+0.328865528 container start 2e371a2f9616261e0dd6006914f213ed9297c204627189457448fc67fb26f821 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, org.label-schema.build-date=20240430, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:41:03.840 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 ceph-osd[58533]: -- 172.21.15.86:0/3701324417 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556111b25680 con 0x556111aef000 2024-05-06T12:41:03.840 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 bash[58513]: 2e371a2f9616261e0dd6006914f213ed9297c204627189457448fc67fb26f821 2024-05-06T12:41:03.840 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 systemd[1]: Started Ceph osd.4 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:41:04.119 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:03 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:41:03.961+0000 7fc2402677c0 -1 Falling back to public interface 2024-05-06T12:41:04.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:03 smithi086 ceph-mon[55252]: pgmap v137: 161 pgs: 7 active+undersized+degraded, 32 peering, 6 stale+active+clean, 12 active+undersized, 104 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 29/627 objects degraded (4.625%) 2024-05-06T12:41:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:04.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:41:03] "GET /metrics HTTP/1.1" 200 38502 "" "Prometheus/2.33.4" 2024-05-06T12:41:04.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[62070]: pgmap v137: 161 pgs: 7 active+undersized+degraded, 32 peering, 6 stale+active+clean, 12 active+undersized, 104 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 29/627 objects degraded (4.625%) 2024-05-06T12:41:04.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[64635]: pgmap v137: 161 pgs: 7 active+undersized+degraded, 32 peering, 6 stale+active+clean, 12 active+undersized, 104 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 29/627 objects degraded (4.625%) 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:04.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:05.199 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:04 smithi086 ceph-mon[55252]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-06T12:41:05.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:04 smithi067 ceph-mon[62070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-06T12:41:05.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:04 smithi067 ceph-mon[64635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-06T12:41:06.186 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:05 smithi086 ceph-mon[55252]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:06.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:05 smithi067 ceph-mon[62070]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:06.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:05 smithi067 ceph-mon[64635]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:06.898 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:41:06.677+0000 7fc2402677c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-05-06T12:41:06.898 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:41:06.708+0000 7fc2402677c0 -1 osd.4 125 log_to_monitors true 2024-05-06T12:41:07.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:06 smithi067 ceph-mon[62070]: from='osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:41:07.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:06 smithi067 ceph-mon[62070]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:41:07.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:06 smithi067 ceph-mon[64635]: from='osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:41:07.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:06 smithi067 ceph-mon[64635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:41:07.308 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:06 smithi086 ceph-mon[55252]: from='osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:41:07.308 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:06 smithi086 ceph-mon[55252]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T12:41:07.619 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:41:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:41:07.362+0000 7fc22d6b6700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: pgmap v139: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: osdmap e128: 8 total, 7 up, 8 in 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: from='osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: from='osd.4 ' entity='osd.4' 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:08.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: pgmap v139: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: osdmap e128: 8 total, 7 up, 8 in 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: from='osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: from='osd.4 ' entity='osd.4' 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:08.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:08.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: pgmap v139: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:08.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T12:41:08.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: osdmap e128: 8 total, 7 up, 8 in 2024-05-06T12:41:08.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: from='osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:08.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:08.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: from='osd.4 ' entity='osd.4' 2024-05-06T12:41:08.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:08.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:09.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:41:09.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[64635]: osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548] boot 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[64635]: osdmap e129: 8 total, 8 up, 8 in 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[62070]: osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548] boot 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[62070]: osdmap e129: 8 total, 8 up, 8 in 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:09.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:09.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:08 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:09.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:08 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:41:09.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:08 smithi086 ceph-mon[55252]: osd.4 [v2:172.21.15.86:6800/690520548,v1:172.21.15.86:6801/690520548] boot 2024-05-06T12:41:09.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:08 smithi086 ceph-mon[55252]: osdmap e129: 8 total, 8 up, 8 in 2024-05-06T12:41:09.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T12:41:09.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:08 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:09.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:09.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:08 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:09.796 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (5m) 37s ago 10m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (4m) 2s ago 9m 50.3M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (4m) 37s ago 9m 53.1M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (4m) 2s ago 14m 451M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (4m) 37s ago 16m 508M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (4m) 37s ago 16m 47.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (3m) 2s ago 15m 35.8M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (3m) 37s ago 15m 38.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (5m) 37s ago 10m 18.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:41:10.472 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (5m) 2s ago 10m 15.8M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (2m) 37s ago 14m 51.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (112s) 37s ago 14m 52.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (77s) 37s ago 13m 40.8M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (42s) 37s ago 13m 14.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (7s) 2s ago 12m 11.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 2e371a2f9616 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (12m) 2s ago 12m 54.5M 2992M 17.2.0 e1d6a67b021e 283936b18db6 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (12m) 2s ago 12m 56.7M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (11m) 2s ago 11m 59.6M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (4m) 2s ago 11m 68.5M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (9m) 37s ago 9m 75.5M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (9m) 2s ago 9m 73.3M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (9m) 37s ago 9m 75.1M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:41:10.473 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (9m) 2s ago 9m 73.2M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:41:10.679 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:10 smithi086 ceph-mon[55252]: pgmap v142: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:10.679 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:10 smithi086 ceph-mon[55252]: osdmap e130: 8 total, 8 up, 8 in 2024-05-06T12:41:10.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:10 smithi067 ceph-mon[62070]: pgmap v142: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:10.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:10 smithi067 ceph-mon[62070]: osdmap e130: 8 total, 8 up, 8 in 2024-05-06T12:41:10.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:10 smithi067 ceph-mon[64635]: pgmap v142: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-05-06T12:41:10.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:10 smithi067 ceph-mon[64635]: osdmap e130: 8 total, 8 up, 8 in 2024-05-06T12:41:10.885 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:41:10.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:10.887 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:41:10.887 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-05-06T12:41:10.887 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 10 2024-05-06T12:41:10.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:41:10.887 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:41:11.230 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:41:11.230 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "10/23 daemons upgraded", 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:41:11.231 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:41:11.648 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN Degraded data redundancy: 56/627 objects degraded (8.931%), 16 pgs degraded 2024-05-06T12:41:11.648 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 56/627 objects degraded (8.931%), 16 pgs degraded 2024-05-06T12:41:11.648 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-05-06T12:41:11.648 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.f is active+undersized+degraded, acting [0,2] 2024-05-06T12:41:11.648 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-05-06T12:41:11.648 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.d is active+undersized+degraded, acting [2,1] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.e is active+undersized+degraded, acting [6,1] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-05-06T12:41:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[62070]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[62070]: from='client.34425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[62070]: from='client.44359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/408538716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[64635]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[64635]: from='client.34425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[64635]: from='client.44359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.650 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:11 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/408538716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:11 smithi086 ceph-mon[55252]: from='client.44347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:11 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:11 smithi086 ceph-mon[55252]: from='client.34425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:11 smithi086 ceph-mon[55252]: from='client.44359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:11.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:11 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/408538716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: pgmap v144: 161 pgs: 16 active+undersized+degraded, 29 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T12:41:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/2685469700' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:41:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:12.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:41:12.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:12 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:12 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:41:12.514+0000 7f337218f700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-05-06T12:41:12.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: pgmap v144: 161 pgs: 16 active+undersized+degraded, 29 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T12:41:12.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:12.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/2685469700' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:41:12.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: pgmap v144: 161 pgs: 16 active+undersized+degraded, 29 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/2685469700' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:41:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:12 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:41:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:41:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:13.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:14.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:41:13] "GET /metrics HTTP/1.1" 200 38501 "" "Prometheus/2.33.4" 2024-05-06T12:41:14.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:14.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[62070]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-05-06T12:41:14.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[62070]: pgmap v145: 161 pgs: 2 active+undersized+degraded, 9 active+undersized, 150 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 9/627 objects degraded (1.435%) 2024-05-06T12:41:14.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 9/627 objects degraded (1.435%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:14.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:14.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[64635]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-05-06T12:41:14.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[64635]: pgmap v145: 161 pgs: 2 active+undersized+degraded, 9 active+undersized, 150 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 9/627 objects degraded (1.435%) 2024-05-06T12:41:14.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:14 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 9/627 objects degraded (1.435%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:14 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:14 smithi086 ceph-mon[55252]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-05-06T12:41:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:14 smithi086 ceph-mon[55252]: pgmap v145: 161 pgs: 2 active+undersized+degraded, 9 active+undersized, 150 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 9/627 objects degraded (1.435%) 2024-05-06T12:41:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:14 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 9/627 objects degraded (1.435%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:15.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:15 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/627 objects degraded (1.435%), 2 pgs degraded) 2024-05-06T12:41:15.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:15 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:41:15.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:15 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/627 objects degraded (1.435%), 2 pgs degraded) 2024-05-06T12:41:15.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:15 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:41:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:15 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/627 objects degraded (1.435%), 2 pgs degraded) 2024-05-06T12:41:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:15 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:41:16.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:16 smithi067 ceph-mon[64635]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:41:16.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:16 smithi067 ceph-mon[62070]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:41:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:16 smithi086 ceph-mon[55252]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:41:18.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:18 smithi067 ceph-mon[64635]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-05-06T12:41:18.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:18 smithi067 ceph-mon[62070]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-05-06T12:41:18.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:18 smithi086 ceph-mon[55252]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-05-06T12:41:20.040 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:19 smithi086 ceph-mon[55252]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:41:20.040 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:20.040 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:20.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:19 smithi067 ceph-mon[64635]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:41:20.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:20.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:20.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:19 smithi067 ceph-mon[62070]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:41:20.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:20.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:21.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:20 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:21.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:20 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:21.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:20 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:22.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:21 smithi086 ceph-mon[55252]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-05-06T12:41:22.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:21 smithi067 ceph-mon[62070]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-05-06T12:41:22.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:21 smithi067 ceph-mon[64635]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-05-06T12:41:24.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:23 smithi086 ceph-mon[55252]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:24.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:23 smithi067 ceph-mon[62070]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:24.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:41:23] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.33.4" 2024-05-06T12:41:24.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:23 smithi067 ceph-mon[64635]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:25.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:25 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:25.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:25 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:25.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:25 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:26.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:26 smithi086 ceph-mon[55252]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:26.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:26 smithi067 ceph-mon[64635]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:26.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:26 smithi067 ceph-mon[62070]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:28.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:28 smithi086 ceph-mon[55252]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:28.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:28 smithi067 ceph-mon[62070]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:28 smithi067 ceph-mon[64635]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:28.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:29.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:29 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:29.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:29 smithi086 ceph-mon[55252]: Upgrade: osd.5 is safe to restart 2024-05-06T12:41:29.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:29 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:29.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:29 smithi067 ceph-mon[62070]: Upgrade: osd.5 is safe to restart 2024-05-06T12:41:29.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:29 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T12:41:29.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:29 smithi067 ceph-mon[64635]: Upgrade: osd.5 is safe to restart 2024-05-06T12:41:30.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[62070]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:30.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[62070]: Upgrade: Updating osd.5 2024-05-06T12:41:30.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:30.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[62070]: Deploying daemon osd.5 on smithi086 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[64635]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[64635]: Upgrade: Updating osd.5 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:30.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:30 smithi067 ceph-mon[64635]: Deploying daemon osd.5 on smithi086 2024-05-06T12:41:30.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:30 smithi086 ceph-mon[55252]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:30.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:30 smithi086 ceph-mon[55252]: Upgrade: Updating osd.5 2024-05-06T12:41:30.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:30 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:30.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:30 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T12:41:30.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:30 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:30.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:30 smithi086 ceph-mon[55252]: Deploying daemon osd.5 on smithi086 2024-05-06T12:41:31.328 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:31 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:31 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:31.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:31 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:32.287 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:32 smithi086 ceph-mon[55252]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:32.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:32 smithi067 ceph-mon[62070]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:32.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:32 smithi067 ceph-mon[64635]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:41:32.542 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:32 smithi086 systemd[1]: Stopping Ceph osd.5 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:41:32.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[29475]: 2024-05-06T12:41:32.546+0000 7fb536d51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:41:32.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[29475]: 2024-05-06T12:41:32.546+0000 7fb536d51700 -1 osd.5 130 *** Got signal Terminated *** 2024-05-06T12:41:32.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[29475]: 2024-05-06T12:41:32.546+0000 7fb536d51700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:41:33.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:33 smithi086 ceph-mon[55252]: osd.5 marked itself down and dead 2024-05-06T12:41:33.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:33 smithi067 ceph-mon[62070]: osd.5 marked itself down and dead 2024-05-06T12:41:33.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:33 smithi067 ceph-mon[64635]: osd.5 marked itself down and dead 2024-05-06T12:41:33.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:33 smithi086 podman[60409]: 2024-05-06 12:41:33.568057245 +0000 UTC m=+1.130053925 container died 283936b18db6c0dcc18ee329a928ce72baa451e0d59a663070f93b4ba91385d6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, GIT_BRANCH=HEAD, architecture=x86_64, distribution-scope=public, vcs-type=git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, version=8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, release=754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-06T12:41:34.112 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:41:33] "GET /metrics HTTP/1.1" 200 38510 "" "Prometheus/2.33.4" 2024-05-06T12:41:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:34 smithi086 ceph-mon[55252]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:34 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:41:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:34 smithi086 ceph-mon[55252]: osdmap e131: 8 total, 7 up, 8 in 2024-05-06T12:41:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:34.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:34.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:33 smithi086 podman[60409]: 2024-05-06 12:41:33.966732183 +0000 UTC m=+1.528728860 container cleanup 283936b18db6c0dcc18ee329a928ce72baa451e0d59a663070f93b4ba91385d6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , release=754, GIT_CLEAN=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=) 2024-05-06T12:41:34.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:33 smithi086 bash[60409]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5 2024-05-06T12:41:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[62070]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:41:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[62070]: osdmap e131: 8 total, 7 up, 8 in 2024-05-06T12:41:34.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:34.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[64635]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:41:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:41:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[64635]: osdmap e131: 8 total, 7 up, 8 in 2024-05-06T12:41:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:34.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:34.581 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:34 smithi086 podman[60425]: 2024-05-06 12:41:34.226625143 +0000 UTC m=+0.656907589 container remove 283936b18db6c0dcc18ee329a928ce72baa451e0d59a663070f93b4ba91385d6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, release=754, version=8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.expose-services=, maintainer=Guillaume Abrioux , RELEASE=HEAD) 2024-05-06T12:41:34.839 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:34 smithi086 podman[60519]: 2024-05-06 12:41:34.483200323 +0000 UTC m=+0.017925600 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:41:34.839 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:34 smithi086 podman[60519]: 2024-05-06 12:41:34.611272562 +0000 UTC m=+0.145997866 container create 47b41e339b17e004e49bd141454dcfe43c19803b85fe891bdb277da95f64b42d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-06T12:41:34.839 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:34 smithi086 podman[60519]: 2024-05-06 12:41:34.838774648 +0000 UTC m=+0.373499906 container init 47b41e339b17e004e49bd141454dcfe43c19803b85fe891bdb277da95f64b42d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-06T12:41:35.119 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:34 smithi086 podman[60519]: 2024-05-06 12:41:34.842158505 +0000 UTC m=+0.376883768 container start 47b41e339b17e004e49bd141454dcfe43c19803b85fe891bdb277da95f64b42d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-05-06T12:41:35.119 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:34 smithi086 podman[60519]: 2024-05-06 12:41:34.938141365 +0000 UTC m=+0.472866640 container attach 47b41e339b17e004e49bd141454dcfe43c19803b85fe891bdb277da95f64b42d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-05-06T12:41:35.425 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:35 smithi086 podman[60519]: 2024-05-06 12:41:35.133479902 +0000 UTC m=+0.668205155 container died 47b41e339b17e004e49bd141454dcfe43c19803b85fe891bdb277da95f64b42d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-70ce953) 2024-05-06T12:41:35.425 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:35 smithi086 ceph-mon[55252]: osdmap e132: 8 total, 7 up, 8 in 2024-05-06T12:41:35.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:35 smithi067 ceph-mon[62070]: osdmap e132: 8 total, 7 up, 8 in 2024-05-06T12:41:35.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:35 smithi067 ceph-mon[64635]: osdmap e132: 8 total, 7 up, 8 in 2024-05-06T12:41:36.119 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:35 smithi086 podman[60544]: 2024-05-06 12:41:35.708452527 +0000 UTC m=+0.570383405 container remove 47b41e339b17e004e49bd141454dcfe43c19803b85fe891bdb277da95f64b42d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-06T12:41:36.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:35 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.5.service: Deactivated successfully. 2024-05-06T12:41:36.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:35 smithi086 systemd[1]: Stopped Ceph osd.5 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:41:36.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:35 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.5.service: Consumed 8.720s CPU time. 2024-05-06T12:41:36.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:35 smithi086 systemd[1]: Starting Ceph osd.5 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:41:36.421 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:36 smithi086 ceph-mon[55252]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17/627 objects degraded (2.711%) 2024-05-06T12:41:36.421 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:36 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:36.421 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 podman[60649]: 2024-05-06 12:41:36.044024688 +0000 UTC m=+0.017754715 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:41:36.422 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 podman[60649]: 2024-05-06 12:41:36.19338467 +0000 UTC m=+0.167114669 container create 0d8721354e9a423b4d09af86a71f5844b063d4af97da6e551b5d7e25b53eeb0b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-70ce953, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-06T12:41:36.422 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 podman[60649]: 2024-05-06 12:41:36.420797518 +0000 UTC m=+0.394527527 container init 0d8721354e9a423b4d09af86a71f5844b063d4af97da6e551b5d7e25b53eeb0b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-06T12:41:36.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:36 smithi067 ceph-mon[62070]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17/627 objects degraded (2.711%) 2024-05-06T12:41:36.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:36 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:36.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:36 smithi067 ceph-mon[64635]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17/627 objects degraded (2.711%) 2024-05-06T12:41:36.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:36 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 17/627 objects degraded (2.711%), 6 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:36.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 podman[60649]: 2024-05-06 12:41:36.424151348 +0000 UTC m=+0.397881349 container start 0d8721354e9a423b4d09af86a71f5844b063d4af97da6e551b5d7e25b53eeb0b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-06T12:41:36.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 podman[60649]: 2024-05-06 12:41:36.512472746 +0000 UTC m=+0.486202760 container attach 0d8721354e9a423b4d09af86a71f5844b063d4af97da6e551b5d7e25b53eeb0b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate[60664]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate[60664]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 bash[60649]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 bash[60649]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 bash[60649]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 bash[60649]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 bash[60649]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 bash[60649]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 bash[60649]: --> ceph-volume raw activate successful for osd ID: 5 2024-05-06T12:41:37.200 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate[60664]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T12:41:37.201 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate[60664]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T12:41:37.201 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate[60664]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-05-06T12:41:37.201 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate[60664]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T12:41:37.201 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate[60664]: --> ceph-volume raw activate successful for osd ID: 5 2024-05-06T12:41:37.201 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:36 smithi086 podman[60649]: 2024-05-06 12:41:36.894224357 +0000 UTC m=+0.867954362 container died 0d8721354e9a423b4d09af86a71f5844b063d4af97da6e551b5d7e25b53eeb0b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, ceph=True) 2024-05-06T12:41:37.804 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 podman[60786]: 2024-05-06 12:41:37.466243388 +0000 UTC m=+0.566138201 container remove 0d8721354e9a423b4d09af86a71f5844b063d4af97da6e551b5d7e25b53eeb0b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-06T12:41:37.804 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 podman[60852]: 2024-05-06 12:41:37.583110401 +0000 UTC m=+0.017488315 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:41:37.804 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 podman[60852]: 2024-05-06 12:41:37.692121216 +0000 UTC m=+0.126499118 container create 13f5da2a25e3c417c05552b6d41c4c5921a5420e4c09128971aa0109e15f1555 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:41:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:38 smithi086 ceph-mon[55252]: pgmap v159: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 17/627 objects degraded (2.711%) 2024-05-06T12:41:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:38.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 podman[60852]: 2024-05-06 12:41:37.90331749 +0000 UTC m=+0.337695406 container init 13f5da2a25e3c417c05552b6d41c4c5921a5420e4c09128971aa0109e15f1555 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-06T12:41:38.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 podman[60852]: 2024-05-06 12:41:37.906552631 +0000 UTC m=+0.340930537 container start 13f5da2a25e3c417c05552b6d41c4c5921a5420e4c09128971aa0109e15f1555 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-05-06T12:41:38.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 ceph-osd[60872]: -- 172.21.15.86:0/1535150458 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ceab09d680 con 0x55ceab066c00 2024-05-06T12:41:38.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 bash[60852]: 13f5da2a25e3c417c05552b6d41c4c5921a5420e4c09128971aa0109e15f1555 2024-05-06T12:41:38.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:37 smithi086 systemd[1]: Started Ceph osd.5 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:41:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:38 smithi067 ceph-mon[62070]: pgmap v159: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 17/627 objects degraded (2.711%) 2024-05-06T12:41:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:38.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:38.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:38 smithi067 ceph-mon[64635]: pgmap v159: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 17/627 objects degraded (2.711%) 2024-05-06T12:41:38.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:38.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:38.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:41:38.457+0000 7fe2847e57c0 -1 Falling back to public interface 2024-05-06T12:41:40.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:40 smithi067 ceph-mon[62070]: pgmap v160: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T12:41:40.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:40 smithi067 ceph-mon[64635]: pgmap v160: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T12:41:40.504 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:40 smithi086 ceph-mon[55252]: pgmap v160: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T12:41:41.458 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:41:41.166+0000 7fe2847e57c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-05-06T12:41:41.458 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:41 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:41:41.188+0000 7fe2847e57c0 -1 osd.5 130 log_to_monitors true 2024-05-06T12:41:41.459 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:41 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:41.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:41 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:41.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:41 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:42.054 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:41:42.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:42 smithi086 ceph-mon[55252]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T12:41:42.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:42 smithi086 ceph-mon[55252]: from='osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:41:42.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:42 smithi086 ceph-mon[55252]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:41:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:42 smithi067 ceph-mon[62070]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T12:41:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:42 smithi067 ceph-mon[62070]: from='osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:41:42.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:42 smithi067 ceph-mon[62070]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:41:42.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:42 smithi067 ceph-mon[64635]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T12:41:42.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:42 smithi067 ceph-mon[64635]: from='osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:41:42.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:42 smithi067 ceph-mon[64635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (5m) 69s ago 10m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (5m) 0s ago 10m 50.0M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (4m) 69s ago 9m 53.1M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (4m) 0s ago 15m 451M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (5m) 69s ago 17m 508M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (4m) 69s ago 17m 47.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (3m) 0s ago 15m 37.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (4m) 69s ago 15m 38.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (5m) 69s ago 11m 18.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (5m) 0s ago 11m 17.8M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:41:42.750 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (2m) 69s ago 15m 51.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (2m) 69s ago 14m 52.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (109s) 69s ago 14m 40.8M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (74s) 69s ago 13m 14.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (39s) 0s ago 13m 51.1M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 2e371a2f9616 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (5s) 0s ago 13m 12.2M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 13f5da2a25e3 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (12m) 0s ago 12m 57.4M 2992M 17.2.0 e1d6a67b021e acb7b473fbf1 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (12m) 0s ago 12m 61.6M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (4m) 0s ago 11m 68.5M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (10m) 69s ago 10m 75.5M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (9m) 0s ago 9m 73.5M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (9m) 69s ago 9m 75.1M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:41:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (9m) 0s ago 9m 73.3M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:41:43.119 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:41:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:41:42.635+0000 7fe271c34700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:41:43.177 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:41:43.177 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:41:43.177 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:41:43.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 10 2024-05-06T12:41:43.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:41:43.179 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='client.44386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: osdmap e133: 8 total, 7 up, 8 in 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:43.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[64635]: from='osd.5 ' entity='osd.5' 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='client.44386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: osdmap e133: 8 total, 7 up, 8 in 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:43.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:43 smithi067 ceph-mon[62070]: from='osd.5 ' entity='osd.5' 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:41:43.522 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "11/23 daemons upgraded", 2024-05-06T12:41:43.523 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:41:43.523 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:41:43.523 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:41:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='client.44386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T12:41:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: osdmap e133: 8 total, 7 up, 8 in 2024-05-06T12:41:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:41:43.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:43.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:43 smithi086 ceph-mon[55252]: from='osd.5 ' entity='osd.5' 2024-05-06T12:41:43.960 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded 2024-05-06T12:41:43.960 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded 2024-05-06T12:41:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-05-06T12:41:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-05-06T12:41:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-05-06T12:41:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.1f is active+undersized+degraded, acting [6,1] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 5.d is active+undersized+degraded, acting [2,7] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-05-06T12:41:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-05-06T12:41:43.962 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:41:43] "GET /metrics HTTP/1.1" 200 38510 "" "Prometheus/2.33.4" 2024-05-06T12:41:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: pgmap v163: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 46 op/s; 71/627 objects degraded (11.324%) 2024-05-06T12:41:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: from='client.44398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1461554523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:44.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191] boot 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: osdmap e134: 8 total, 8 up, 8 in 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/631342950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: pgmap v163: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 46 op/s; 71/627 objects degraded (11.324%) 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: from='client.44398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1461554523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:44.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:41:44.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191] boot 2024-05-06T12:41:44.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: osdmap e134: 8 total, 8 up, 8 in 2024-05-06T12:41:44.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:41:44.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:44.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:44.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:44 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/631342950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: pgmap v163: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 46 op/s; 71/627 objects degraded (11.324%) 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: from='client.44398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1461554523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: osd.5 [v2:172.21.15.86:6808/1458866191,v1:172.21.15.86:6809/1458866191] boot 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: osdmap e134: 8 total, 8 up, 8 in 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:44.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:44 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/631342950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:41:45.422 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:45 smithi086 ceph-mon[55252]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:45.422 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:45 smithi086 ceph-mon[55252]: osdmap e135: 8 total, 8 up, 8 in 2024-05-06T12:41:45.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:45 smithi067 ceph-mon[62070]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:45.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:45 smithi067 ceph-mon[62070]: osdmap e135: 8 total, 8 up, 8 in 2024-05-06T12:41:45.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:45 smithi067 ceph-mon[64635]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:41:45.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:45 smithi067 ceph-mon[64635]: osdmap e135: 8 total, 8 up, 8 in 2024-05-06T12:41:46.446 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:46 smithi086 ceph-mon[55252]: pgmap v166: 161 pgs: 1 active+recovering+degraded, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 33 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s; 67/627 objects degraded (10.686%); 0 B/s, 0 objects/s recovering 2024-05-06T12:41:46.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:46 smithi067 ceph-mon[62070]: pgmap v166: 161 pgs: 1 active+recovering+degraded, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 33 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s; 67/627 objects degraded (10.686%); 0 B/s, 0 objects/s recovering 2024-05-06T12:41:46.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:46 smithi067 ceph-mon[64635]: pgmap v166: 161 pgs: 1 active+recovering+degraded, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 33 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s; 67/627 objects degraded (10.686%); 0 B/s, 0 objects/s recovering 2024-05-06T12:41:47.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:47 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:41:47.456+0000 7f337218f700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-06T12:41:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: pgmap v167: 161 pgs: 1 active+recovering+degraded, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 33 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s; 67/627 objects degraded (10.686%); 0 B/s, 0 objects/s recovering 2024-05-06T12:41:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:41:48.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:48 smithi086 ceph-mon[55252]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-06T12:41:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: pgmap v167: 161 pgs: 1 active+recovering+degraded, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 33 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s; 67/627 objects degraded (10.686%); 0 B/s, 0 objects/s recovering 2024-05-06T12:41:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:41:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[62070]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: pgmap v167: 161 pgs: 1 active+recovering+degraded, 16 active+undersized+degraded, 1 active+recovery_wait+degraded, 33 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s; 67/627 objects degraded (10.686%); 0 B/s, 0 objects/s recovering 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:41:48.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:41:48.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:48 smithi067 ceph-mon[64635]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-06T12:41:49.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:49 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 67/627 objects degraded (10.686%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:49.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:49.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:49.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:49 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 67/627 objects degraded (10.686%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:49.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:49.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:49 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 67/627 objects degraded (10.686%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T12:41:49.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:41:49.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:41:50.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:50 smithi086 ceph-mon[55252]: pgmap v168: 161 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 159 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 170 op/s; 8/627 objects degraded (1.276%); 0 B/s, 6 objects/s recovering 2024-05-06T12:41:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:50 smithi067 ceph-mon[62070]: pgmap v168: 161 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 159 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 170 op/s; 8/627 objects degraded (1.276%); 0 B/s, 6 objects/s recovering 2024-05-06T12:41:50.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:50 smithi067 ceph-mon[64635]: pgmap v168: 161 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 159 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 170 op/s; 8/627 objects degraded (1.276%); 0 B/s, 6 objects/s recovering 2024-05-06T12:41:51.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:51 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:51.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:51 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:51.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:51 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:41:52.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:52 smithi086 ceph-mon[55252]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T12:41:52.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:52 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/627 objects degraded (1.276%), 2 pgs degraded) 2024-05-06T12:41:52.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:52 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:41:52.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:52 smithi067 ceph-mon[62070]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T12:41:52.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:52 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/627 objects degraded (1.276%), 2 pgs degraded) 2024-05-06T12:41:52.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:52 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:41:52.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:52 smithi067 ceph-mon[64635]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T12:41:52.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:52 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/627 objects degraded (1.276%), 2 pgs degraded) 2024-05-06T12:41:52.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:52 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:41:54.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:41:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:41:53] "GET /metrics HTTP/1.1" 200 38519 "" "Prometheus/2.33.4" 2024-05-06T12:41:54.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:54 smithi086 ceph-mon[55252]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s; 0 B/s, 5 objects/s recovering 2024-05-06T12:41:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:54 smithi067 ceph-mon[62070]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s; 0 B/s, 5 objects/s recovering 2024-05-06T12:41:54.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:54 smithi067 ceph-mon[64635]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s; 0 B/s, 5 objects/s recovering 2024-05-06T12:41:56.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:56 smithi086 ceph-mon[55252]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 181 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:41:56.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:56 smithi067 ceph-mon[62070]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 181 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:41:56.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:56 smithi067 ceph-mon[64635]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 181 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:41:58.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:41:58 smithi086 ceph-mon[55252]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:41:58.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:41:58 smithi067 ceph-mon[62070]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:41:58.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:41:58 smithi067 ceph-mon[64635]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:42:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:00 smithi067 ceph-mon[62070]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:42:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:00 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:00.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:00 smithi067 ceph-mon[64635]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:42:00.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:00 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:00 smithi086 ceph-mon[55252]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 180 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T12:42:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:00 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:01.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:01 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:01.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:01 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:01.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:01 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:02.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:02 smithi086 ceph-mon[55252]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s; 0 B/s, 0 objects/s recovering 2024-05-06T12:42:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:02 smithi067 ceph-mon[62070]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s; 0 B/s, 0 objects/s recovering 2024-05-06T12:42:02.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:02 smithi067 ceph-mon[64635]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s; 0 B/s, 0 objects/s recovering 2024-05-06T12:42:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:03 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:42:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:03 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:42:03.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:03 smithi086 ceph-mon[55252]: Upgrade: osd.6 is safe to restart 2024-05-06T12:42:03.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:03 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:42:03.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:03 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:42:03.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:03 smithi067 ceph-mon[64635]: Upgrade: osd.6 is safe to restart 2024-05-06T12:42:03.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:03 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:42:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:03 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T12:42:03.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:03 smithi067 ceph-mon[62070]: Upgrade: osd.6 is safe to restart 2024-05-06T12:42:04.067 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:42:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:42:03] "GET /metrics HTTP/1.1" 200 38519 "" "Prometheus/2.33.4" 2024-05-06T12:42:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:04 smithi086 ceph-mon[55252]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-05-06T12:42:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:04.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:04 smithi067 ceph-mon[62070]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-05-06T12:42:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:04.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:04 smithi067 ceph-mon[64635]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-05-06T12:42:04.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:04.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:05 smithi086 ceph-mon[55252]: Upgrade: Updating osd.6 2024-05-06T12:42:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:05.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:42:05.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:05.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:05 smithi086 ceph-mon[55252]: Deploying daemon osd.6 on smithi086 2024-05-06T12:42:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[62070]: Upgrade: Updating osd.6 2024-05-06T12:42:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:42:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:05.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[62070]: Deploying daemon osd.6 on smithi086 2024-05-06T12:42:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[64635]: Upgrade: Updating osd.6 2024-05-06T12:42:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T12:42:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:05.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:05 smithi067 ceph-mon[64635]: Deploying daemon osd.6 on smithi086 2024-05-06T12:42:06.367 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:06 smithi086 ceph-mon[55252]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-05-06T12:42:06.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:06 smithi067 ceph-mon[62070]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-05-06T12:42:06.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:06 smithi067 ceph-mon[64635]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-05-06T12:42:07.748 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:07 smithi086 systemd[1]: Stopping Ceph osd.6 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:42:08.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[32364]: 2024-05-06T12:42:07.745+0000 7f9b49f31700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:42:08.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[32364]: 2024-05-06T12:42:07.745+0000 7f9b49f31700 -1 osd.6 135 *** Got signal Terminated *** 2024-05-06T12:42:08.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[32364]: 2024-05-06T12:42:07.745+0000 7f9b49f31700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:42:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:08 smithi086 ceph-mon[55252]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-05-06T12:42:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:08 smithi086 ceph-mon[55252]: osd.6 marked itself down and dead 2024-05-06T12:42:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:08 smithi067 ceph-mon[64635]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-05-06T12:42:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:08 smithi067 ceph-mon[64635]: osd.6 marked itself down and dead 2024-05-06T12:42:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:08 smithi067 ceph-mon[62070]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-05-06T12:42:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:08 smithi067 ceph-mon[62070]: osd.6 marked itself down and dead 2024-05-06T12:42:08.998 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:08 smithi086 podman[62749]: 2024-05-06 12:42:08.696097544 +0000 UTC m=+1.097465902 container died acb7b473fbf1d6e2a9a1615f7c18788053925333b6d01f8c732e94c3388b69c7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, RELEASE=HEAD, vendor=Red Hat, Inc., vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-06T12:42:09.340 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:09 smithi086 podman[62749]: 2024-05-06 12:42:09.095947124 +0000 UTC m=+1.497315483 container cleanup acb7b473fbf1d6e2a9a1615f7c18788053925333b6d01f8c732e94c3388b69c7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., name=centos-stream, version=8, GIT_CLEAN=True, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-06T12:42:09.340 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:09 smithi086 bash[62749]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6 2024-05-06T12:42:09.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:09 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:42:09.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:09 smithi086 ceph-mon[55252]: osdmap e136: 8 total, 7 up, 8 in 2024-05-06T12:42:09.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:09 smithi086 ceph-mon[55252]: osdmap e137: 8 total, 7 up, 8 in 2024-05-06T12:42:09.619 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:09 smithi086 podman[62767]: 2024-05-06 12:42:09.339335782 +0000 UTC m=+0.645533078 container remove acb7b473fbf1d6e2a9a1615f7c18788053925333b6d01f8c732e94c3388b69c7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, CEPH_POINT_RELEASE=-17.2.0, version=8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, release=754, vendor=Red Hat, Inc., vcs-type=git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, name=centos-stream) 2024-05-06T12:42:09.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:09 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:42:09.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:09 smithi067 ceph-mon[62070]: osdmap e136: 8 total, 7 up, 8 in 2024-05-06T12:42:09.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:09 smithi067 ceph-mon[62070]: osdmap e137: 8 total, 7 up, 8 in 2024-05-06T12:42:09.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:09 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:42:09.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:09 smithi067 ceph-mon[64635]: osdmap e136: 8 total, 7 up, 8 in 2024-05-06T12:42:09.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:09 smithi067 ceph-mon[64635]: osdmap e137: 8 total, 7 up, 8 in 2024-05-06T12:42:09.935 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:09 smithi086 podman[62853]: 2024-05-06 12:42:09.573391983 +0000 UTC m=+0.018266398 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:42:09.935 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:09 smithi086 podman[62853]: 2024-05-06 12:42:09.715631967 +0000 UTC m=+0.160506383 container create a403008283e085320c293f054e75ef310bfabba773231f7d123a943320674a31 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, GIT_CLEAN=True, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-06T12:42:09.935 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:09 smithi086 podman[62853]: 2024-05-06 12:42:09.934778316 +0000 UTC m=+0.379652732 container init a403008283e085320c293f054e75ef310bfabba773231f7d123a943320674a31 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:42:10.231 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:09 smithi086 podman[62853]: 2024-05-06 12:42:09.938097911 +0000 UTC m=+0.382972328 container start a403008283e085320c293f054e75ef310bfabba773231f7d123a943320674a31 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, ceph=True) 2024-05-06T12:42:10.232 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:10 smithi086 podman[62853]: 2024-05-06 12:42:10.034658843 +0000 UTC m=+0.479533262 container attach a403008283e085320c293f054e75ef310bfabba773231f7d123a943320674a31 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=) 2024-05-06T12:42:10.597 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:10 smithi086 ceph-mon[55252]: pgmap v179: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:42:10.597 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:10 smithi086 podman[62853]: 2024-05-06 12:42:10.230922698 +0000 UTC m=+0.675797121 container died a403008283e085320c293f054e75ef310bfabba773231f7d123a943320674a31 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:42:10.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:10 smithi067 ceph-mon[64635]: pgmap v179: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:42:10.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:10 smithi067 ceph-mon[62070]: pgmap v179: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:42:11.250 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:10 smithi086 podman[62880]: 2024-05-06 12:42:10.871790919 +0000 UTC m=+0.635416193 container remove a403008283e085320c293f054e75ef310bfabba773231f7d123a943320674a31 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-05-06T12:42:11.250 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:10 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.6.service: Deactivated successfully. 2024-05-06T12:42:11.251 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:10 smithi086 systemd[1]: Stopped Ceph osd.6 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:42:11.251 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:10 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.6.service: Consumed 8.151s CPU time. 2024-05-06T12:42:11.251 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:11 smithi086 systemd[1]: Starting Ceph osd.6 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:42:11.576 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:11 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:11.576 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:11 smithi086 podman[62984]: 2024-05-06 12:42:11.213523178 +0000 UTC m=+0.017847014 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:42:11.576 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:11 smithi086 podman[62984]: 2024-05-06 12:42:11.348400764 +0000 UTC m=+0.152724589 container create 6afc78648be164cc872f64c2bd7233c5003df717a47a2ad51915a85e7b2cd098 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-05-06T12:42:11.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:11 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:11.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:11 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:11.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:11 smithi086 podman[62984]: 2024-05-06 12:42:11.575882502 +0000 UTC m=+0.380206328 container init 6afc78648be164cc872f64c2bd7233c5003df717a47a2ad51915a85e7b2cd098 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-05-06T12:42:11.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:11 smithi086 podman[62984]: 2024-05-06 12:42:11.578983971 +0000 UTC m=+0.383307805 container start 6afc78648be164cc872f64c2bd7233c5003df717a47a2ad51915a85e7b2cd098 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-06T12:42:11.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:11 smithi086 podman[62984]: 2024-05-06 12:42:11.659100683 +0000 UTC m=+0.463424510 container attach 6afc78648be164cc872f64c2bd7233c5003df717a47a2ad51915a85e7b2cd098 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-06T12:42:12.355 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:12 smithi086 ceph-mon[55252]: pgmap v181: 161 pgs: 4 active+undersized+degraded, 12 stale+active+clean, 6 active+undersized, 139 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 16/627 objects degraded (2.552%) 2024-05-06T12:42:12.355 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:12 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 16/627 objects degraded (2.552%), 4 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate[62999]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate[62999]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 bash[62984]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 bash[62984]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 bash[62984]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 bash[62984]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 bash[62984]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 bash[62984]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 bash[62984]: --> ceph-volume raw activate successful for osd ID: 6 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate[62999]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate[62999]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate[62999]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-05-06T12:42:12.356 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate[62999]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T12:42:12.357 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate[62999]: --> ceph-volume raw activate successful for osd ID: 6 2024-05-06T12:42:12.357 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 podman[62984]: 2024-05-06 12:42:12.036941019 +0000 UTC m=+0.841264845 container died 6afc78648be164cc872f64c2bd7233c5003df717a47a2ad51915a85e7b2cd098 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:42:12.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:12 smithi067 ceph-mon[62070]: pgmap v181: 161 pgs: 4 active+undersized+degraded, 12 stale+active+clean, 6 active+undersized, 139 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 16/627 objects degraded (2.552%) 2024-05-06T12:42:12.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:12 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 16/627 objects degraded (2.552%), 4 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:12.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:12 smithi067 ceph-mon[64635]: pgmap v181: 161 pgs: 4 active+undersized+degraded, 12 stale+active+clean, 6 active+undersized, 139 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 16/627 objects degraded (2.552%) 2024-05-06T12:42:12.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:12 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 16/627 objects degraded (2.552%), 4 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:12.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 podman[63117]: 2024-05-06 12:42:12.612813395 +0000 UTC m=+0.565939797 container remove 6afc78648be164cc872f64c2bd7233c5003df717a47a2ad51915a85e7b2cd098 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-06T12:42:12.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 podman[63183]: 2024-05-06 12:42:12.727777585 +0000 UTC m=+0.017869543 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:42:12.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:12 smithi086 podman[63183]: 2024-05-06 12:42:12.838948215 +0000 UTC m=+0.129040174 container create c803277a9f3db9e4c7df13d168711a4a3d5473820e88a0c6afbd8420fa78a9e2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-06T12:42:13.619 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:13 smithi086 podman[63183]: 2024-05-06 12:42:13.241634218 +0000 UTC m=+0.531726175 container init c803277a9f3db9e4c7df13d168711a4a3d5473820e88a0c6afbd8420fa78a9e2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-06T12:42:13.620 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:13 smithi086 podman[63183]: 2024-05-06 12:42:13.244677285 +0000 UTC m=+0.534769250 container start c803277a9f3db9e4c7df13d168711a4a3d5473820e88a0c6afbd8420fa78a9e2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-70ce953) 2024-05-06T12:42:13.620 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:13 smithi086 ceph-osd[63204]: -- 172.21.15.86:0/3957537886 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c8fe79b680 con 0x55c8fe765000 2024-05-06T12:42:13.620 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:13 smithi086 bash[63183]: c803277a9f3db9e4c7df13d168711a4a3d5473820e88a0c6afbd8420fa78a9e2 2024-05-06T12:42:13.620 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:13 smithi086 systemd[1]: Started Ceph osd.6 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:42:14.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:42:13.787+0000 7f29147737c0 -1 Falling back to public interface 2024-05-06T12:42:14.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:42:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:42:13] "GET /metrics HTTP/1.1" 200 38514 "" "Prometheus/2.33.4" 2024-05-06T12:42:14.336 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:42:14.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:14 smithi086 ceph-mon[55252]: pgmap v182: 161 pgs: 17 active+undersized+degraded, 1 stale+active+clean, 31 active+undersized, 112 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-05-06T12:42:14.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:14 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:14.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:14 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:14.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:14 smithi067 ceph-mon[62070]: pgmap v182: 161 pgs: 17 active+undersized+degraded, 1 stale+active+clean, 31 active+undersized, 112 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-05-06T12:42:14.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:14 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:14.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:14 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:14.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:14 smithi067 ceph-mon[64635]: pgmap v182: 161 pgs: 17 active+undersized+degraded, 1 stale+active+clean, 31 active+undersized, 112 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-05-06T12:42:14.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:14 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:14.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:14 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (6m) 101s ago 11m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (6m) 32s ago 11m 50.0M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (5m) 101s ago 10m 53.1M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (5m) 32s ago 15m 451M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (5m) 101s ago 17m 508M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (5m) 101s ago 17m 47.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:42:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (4m) 32s ago 16m 37.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (4m) 101s ago 16m 38.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (6m) 101s ago 12m 18.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (6m) 32s ago 12m 17.8M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (3m) 101s ago 15m 51.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (2m) 101s ago 15m 52.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (2m) 101s ago 14m 40.8M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (107s) 101s ago 14m 14.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (71s) 32s ago 14m 51.1M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 2e371a2f9616 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (37s) 32s ago 13m 12.2M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 13f5da2a25e3 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 starting - - - 2992M 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (12m) 32s ago 12m 61.6M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (5m) 32s ago 12m 68.5M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (10m) 101s ago 10m 75.5M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (10m) 32s ago 10m 73.5M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:42:15.021 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (10m) 101s ago 10m 75.1M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:42:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (10m) 32s ago 10m 73.3M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:42:15.432 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 6 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:15.433 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:42:15.434 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-05-06T12:42:15.434 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 11 2024-05-06T12:42:15.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:42:15.434 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:42:15.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:15 smithi067 ceph-mon[62070]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:15.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:15 smithi067 ceph-mon[64635]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:42:15.775 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:42:15.776 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "11/23 daemons upgraded", 2024-05-06T12:42:15.776 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:42:15.776 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:42:15.776 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:42:15.842 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:15 smithi086 ceph-mon[55252]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:16.191 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded 2024-05-06T12:42:16.191 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] OSD_DOWN: 1 osds down 2024-05-06T12:42:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: osd.6 (root=default,host=smithi086) is down 2024-05-06T12:42:16.191 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded 2024-05-06T12:42:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-05-06T12:42:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-05-06T12:42:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-05-06T12:42:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-05-06T12:42:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[64635]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[64635]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:16.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[64635]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:16.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/2034333035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:16.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/2153225248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:42:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[62070]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[62070]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[62070]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/2034333035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:16.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:16 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/2153225248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:42:16.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:42:16.515+0000 7f29147737c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-05-06T12:42:16.735 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:42:16.546+0000 7f29147737c0 -1 osd.6 135 log_to_monitors true 2024-05-06T12:42:16.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:16 smithi086 ceph-mon[55252]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:16.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:16 smithi086 ceph-mon[55252]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:16.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:16 smithi086 ceph-mon[55252]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:16.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:16 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/2034333035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:16.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:16 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/2153225248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:42:17.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:17 smithi086 ceph-mon[55252]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:17.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:17 smithi086 ceph-mon[55252]: from='osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:42:17.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:17 smithi086 ceph-mon[55252]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:42:17.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:17 smithi067 ceph-mon[64635]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:17.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:17 smithi067 ceph-mon[64635]: from='osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:42:17.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:17 smithi067 ceph-mon[64635]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:42:17.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:17 smithi067 ceph-mon[62070]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:17.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:17 smithi067 ceph-mon[62070]: from='osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:42:17.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:17 smithi067 ceph-mon[62070]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T12:42:18.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:42:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:42:17.854+0000 7f2901bc2700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:42:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: pgmap v184: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T12:42:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: from='osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: osdmap e138: 8 total, 7 up, 8 in 2024-05-06T12:42:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: from='osd.6 ' entity='osd.6' 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: pgmap v184: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: from='osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: osdmap e138: 8 total, 7 up, 8 in 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: from='osd.6 ' entity='osd.6' 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:18.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:18.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: pgmap v184: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:18.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T12:42:18.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: from='osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:18.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: osdmap e138: 8 total, 7 up, 8 in 2024-05-06T12:42:18.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:18.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: from='osd.6 ' entity='osd.6' 2024-05-06T12:42:18.872 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:18.872 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918] boot 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: osdmap e139: 8 total, 8 up, 8 in 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:42:19.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.687 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:42:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918] boot 2024-05-06T12:42:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: osdmap e139: 8 total, 8 up, 8 in 2024-05-06T12:42:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:42:19.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:19.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:19.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:42:19.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: osd.6 [v2:172.21.15.86:6816/249109918,v1:172.21.15.86:6817/249109918] boot 2024-05-06T12:42:19.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: osdmap e139: 8 total, 8 up, 8 in 2024-05-06T12:42:19.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T12:42:19.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:19.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:20.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:20 smithi086 ceph-mon[55252]: pgmap v186: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:20.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:20 smithi086 ceph-mon[55252]: osdmap e140: 8 total, 8 up, 8 in 2024-05-06T12:42:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:20 smithi067 ceph-mon[62070]: pgmap v186: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:20 smithi067 ceph-mon[62070]: osdmap e140: 8 total, 8 up, 8 in 2024-05-06T12:42:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:20 smithi067 ceph-mon[64635]: pgmap v186: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T12:42:20.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:20 smithi067 ceph-mon[64635]: osdmap e140: 8 total, 8 up, 8 in 2024-05-06T12:42:21.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:21 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:21.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:21 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:21.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:21 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:22.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:22 smithi086 ceph-mon[55252]: pgmap v189: 161 pgs: 9 peering, 14 active+undersized+degraded, 30 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-05-06T12:42:22.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:22 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:22 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:22 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:22.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:22 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:42:22.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:22 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.725 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:42:22 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:42:22.502+0000 7f337218f700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T12:42:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[62070]: pgmap v189: 161 pgs: 9 peering, 14 active+undersized+degraded, 30 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-05-06T12:42:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:22.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:42:22.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[64635]: pgmap v189: 161 pgs: 9 peering, 14 active+undersized+degraded, 30 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-05-06T12:42:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:42:22.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:22 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:23.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[62070]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:23.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:23 smithi067 ceph-mon[64635]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T12:42:23.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:42:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:23 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:23.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:23 smithi086 ceph-mon[55252]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T12:42:24.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:42:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:42:23] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.33.4" 2024-05-06T12:42:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:24 smithi067 ceph-mon[62070]: pgmap v190: 161 pgs: 9 peering, 1 active+undersized+degraded, 3 active+undersized, 148 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 9/627 objects degraded (1.435%) 2024-05-06T12:42:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:24 smithi067 ceph-mon[62070]: Health check update: Degraded data redundancy: 9/627 objects degraded (1.435%), 1 pg degraded (PG_DEGRADED) 2024-05-06T12:42:24.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:24 smithi067 ceph-mon[64635]: pgmap v190: 161 pgs: 9 peering, 1 active+undersized+degraded, 3 active+undersized, 148 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 9/627 objects degraded (1.435%) 2024-05-06T12:42:24.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:24 smithi067 ceph-mon[64635]: Health check update: Degraded data redundancy: 9/627 objects degraded (1.435%), 1 pg degraded (PG_DEGRADED) 2024-05-06T12:42:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:24 smithi086 ceph-mon[55252]: pgmap v190: 161 pgs: 9 peering, 1 active+undersized+degraded, 3 active+undersized, 148 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 9/627 objects degraded (1.435%) 2024-05-06T12:42:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:24 smithi086 ceph-mon[55252]: Health check update: Degraded data redundancy: 9/627 objects degraded (1.435%), 1 pg degraded (PG_DEGRADED) 2024-05-06T12:42:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:25 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/627 objects degraded (1.435%), 1 pg degraded) 2024-05-06T12:42:25.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:25 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:42:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:25 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/627 objects degraded (1.435%), 1 pg degraded) 2024-05-06T12:42:25.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:25 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:42:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:25 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/627 objects degraded (1.435%), 1 pg degraded) 2024-05-06T12:42:25.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:25 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:42:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:26 smithi086 ceph-mon[55252]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:42:26.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:26 smithi067 ceph-mon[62070]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:42:26.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:26 smithi067 ceph-mon[64635]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:42:28.777 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:28 smithi067 ceph-mon[62070]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-05-06T12:42:28.777 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:28 smithi067 ceph-mon[64635]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-05-06T12:42:28.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:28 smithi086 ceph-mon[55252]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-05-06T12:42:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:30 smithi086 ceph-mon[55252]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-05-06T12:42:30.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:30 smithi067 ceph-mon[62070]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-05-06T12:42:30.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:30 smithi067 ceph-mon[64635]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-05-06T12:42:31.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:31 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:31.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:31 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:31.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:31 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:32.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:32 smithi086 ceph-mon[55252]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-05-06T12:42:32.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:32 smithi067 ceph-mon[62070]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-05-06T12:42:32.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:32 smithi067 ceph-mon[64635]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-05-06T12:42:34.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:42:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:42:33] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.33.4" 2024-05-06T12:42:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:34 smithi086 ceph-mon[55252]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:42:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:34.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[64635]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:42:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[62070]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:42:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:34.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:36 smithi086 ceph-mon[55252]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:36.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:36 smithi067 ceph-mon[62070]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:36.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:36 smithi067 ceph-mon[64635]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:38.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:38 smithi086 ceph-mon[55252]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:38.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:38 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:38.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:38 smithi086 ceph-mon[55252]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:38.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:38 smithi086 ceph-mon[55252]: Upgrade: osd.7 is safe to restart 2024-05-06T12:42:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[62070]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[62070]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:38.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[62070]: Upgrade: osd.7 is safe to restart 2024-05-06T12:42:38.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[64635]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:38.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:38.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[64635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T12:42:38.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:38 smithi067 ceph-mon[64635]: Upgrade: osd.7 is safe to restart 2024-05-06T12:42:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:40 smithi086 ceph-mon[55252]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:40 smithi086 ceph-mon[55252]: Upgrade: Updating osd.7 2024-05-06T12:42:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:40 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:40 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:42:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:40 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:40 smithi086 ceph-mon[55252]: Deploying daemon osd.7 on smithi086 2024-05-06T12:42:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[62070]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[62070]: Upgrade: Updating osd.7 2024-05-06T12:42:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[62070]: Deploying daemon osd.7 on smithi086 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[64635]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[64635]: Upgrade: Updating osd.7 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:40.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:40 smithi067 ceph-mon[64635]: Deploying daemon osd.7 on smithi086 2024-05-06T12:42:41.595 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:41 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:41.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:41 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:41.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:41 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:42.481 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:42 smithi086 ceph-mon[55252]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:42.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:42 smithi067 ceph-mon[64635]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:42.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:42 smithi067 ceph-mon[62070]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:42:42.869 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:42 smithi086 systemd[1]: Stopping Ceph osd.7 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:42:43.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[35280]: 2024-05-06T12:42:42.944+0000 7f0eb1fc9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:42:43.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[35280]: 2024-05-06T12:42:42.944+0000 7f0eb1fc9700 -1 osd.7 140 *** Got signal Terminated *** 2024-05-06T12:42:43.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:42 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[35280]: 2024-05-06T12:42:42.944+0000 7f0eb1fc9700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:42:43.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:43 smithi067 ceph-mon[64635]: osd.7 marked itself down and dead 2024-05-06T12:42:43.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:43 smithi067 ceph-mon[62070]: osd.7 marked itself down and dead 2024-05-06T12:42:43.764 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:43 smithi086 ceph-mon[55252]: osd.7 marked itself down and dead 2024-05-06T12:42:44.054 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:43 smithi086 podman[65069]: 2024-05-06 12:42:43.763995486 +0000 UTC m=+0.954839272 container died b98d9a542c752ad3f3f2f3db208bea7caccb47517b9de0ec708157d63d872f0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, architecture=x86_64, vcs-type=git, GIT_CLEAN=True, name=centos-stream, io.openshift.expose-services=, version=8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container) 2024-05-06T12:42:44.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:42:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:42:43] "GET /metrics HTTP/1.1" 200 38532 "" "Prometheus/2.33.4" 2024-05-06T12:42:44.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:44 smithi086 podman[65069]: 2024-05-06 12:42:44.152412024 +0000 UTC m=+1.343255810 container cleanup b98d9a542c752ad3f3f2f3db208bea7caccb47517b9de0ec708157d63d872f0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , architecture=x86_64, vcs-type=git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, release=754, io.openshift.expose-services=, version=8, GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-06T12:42:44.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:44 smithi086 bash[65069]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7 2024-05-06T12:42:44.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[62070]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:44.788 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[62070]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:42:44.789 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[62070]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-06T12:42:44.789 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[62070]: osdmap e141: 8 total, 7 up, 8 in 2024-05-06T12:42:44.789 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[64635]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:44.789 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[64635]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:42:44.789 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[64635]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-06T12:42:44.789 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:44 smithi067 ceph-mon[64635]: osdmap e141: 8 total, 7 up, 8 in 2024-05-06T12:42:44.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:44 smithi086 ceph-mon[55252]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:44.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:44 smithi086 ceph-mon[55252]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T12:42:44.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:44 smithi086 ceph-mon[55252]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-06T12:42:44.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:44 smithi086 ceph-mon[55252]: osdmap e141: 8 total, 7 up, 8 in 2024-05-06T12:42:44.870 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:44 smithi086 podman[65087]: 2024-05-06 12:42:44.445571058 +0000 UTC m=+0.684292137 container remove b98d9a542c752ad3f3f2f3db208bea7caccb47517b9de0ec708157d63d872f0d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, release=754, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vcs-type=git, io.buildah.version=1.19.8, name=centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream) 2024-05-06T12:42:45.250 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:44 smithi086 podman[65180]: 2024-05-06 12:42:44.835707526 +0000 UTC m=+0.018222111 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:42:45.250 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:44 smithi086 podman[65180]: 2024-05-06 12:42:44.954804638 +0000 UTC m=+0.137319210 container create 92242371c09fbfddb37dcced6e993fa140cdf340d6d9f4e2ba5d062c043dde54 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:42:45.250 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:45 smithi086 podman[65180]: 2024-05-06 12:42:45.157691573 +0000 UTC m=+0.340206146 container init 92242371c09fbfddb37dcced6e993fa140cdf340d6d9f4e2ba5d062c043dde54 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-06T12:42:45.250 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:45 smithi086 podman[65180]: 2024-05-06 12:42:45.161016213 +0000 UTC m=+0.343530795 container start 92242371c09fbfddb37dcced6e993fa140cdf340d6d9f4e2ba5d062c043dde54 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:42:45.250 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:45 smithi086 podman[65180]: 2024-05-06 12:42:45.248961766 +0000 UTC m=+0.431476339 container attach 92242371c09fbfddb37dcced6e993fa140cdf340d6d9f4e2ba5d062c043dde54 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-05-06T12:42:45.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:45 smithi086 ceph-mon[55252]: osdmap e142: 8 total, 7 up, 8 in 2024-05-06T12:42:45.620 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:45 smithi086 podman[65180]: 2024-05-06 12:42:45.449918876 +0000 UTC m=+0.632433441 container died 92242371c09fbfddb37dcced6e993fa140cdf340d6d9f4e2ba5d062c043dde54 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, RELEASE=squid-70ce953) 2024-05-06T12:42:45.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:45 smithi067 ceph-mon[62070]: osdmap e142: 8 total, 7 up, 8 in 2024-05-06T12:42:45.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:45 smithi067 ceph-mon[64635]: osdmap e142: 8 total, 7 up, 8 in 2024-05-06T12:42:46.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 podman[65206]: 2024-05-06 12:42:46.044146099 +0000 UTC m=+0.585990510 container remove 92242371c09fbfddb37dcced6e993fa140cdf340d6d9f4e2ba5d062c043dde54 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-06T12:42:46.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.7.service: Deactivated successfully. 2024-05-06T12:42:46.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 systemd[1]: Stopped Ceph osd.7 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:42:46.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.7.service: Consumed 10.618s CPU time. 2024-05-06T12:42:46.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 systemd[1]: Starting Ceph osd.7 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:42:46.565 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:42:46.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:46 smithi067 ceph-mon[62070]: pgmap v203: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:46.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:46 smithi067 ceph-mon[64635]: pgmap v203: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:46.731 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:46 smithi086 ceph-mon[55252]: pgmap v203: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:42:46.731 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 podman[65312]: 2024-05-06 12:42:46.38966141 +0000 UTC m=+0.017763101 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:42:46.731 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 podman[65312]: 2024-05-06 12:42:46.511924404 +0000 UTC m=+0.140026095 container create 35ba4c1826a835a2328c848733a4370213dfd4518f8a3635e302256acbd6e55c (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-06T12:42:46.731 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 podman[65312]: 2024-05-06 12:42:46.73092175 +0000 UTC m=+0.359023441 container init 35ba4c1826a835a2328c848733a4370213dfd4518f8a3635e302256acbd6e55c (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:42:47.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 podman[65312]: 2024-05-06 12:42:46.734108374 +0000 UTC m=+0.362210076 container start 35ba4c1826a835a2328c848733a4370213dfd4518f8a3635e302256acbd6e55c (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-05-06T12:42:47.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:46 smithi086 podman[65312]: 2024-05-06 12:42:46.822386023 +0000 UTC m=+0.450487726 container attach 35ba4c1826a835a2328c848733a4370213dfd4518f8a3635e302256acbd6e55c (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:42:47.244 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:42:47.244 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (6m) 2m ago 11m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:42:47.244 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (6m) 29s ago 11m 50.2M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (6m) 2m ago 10m 53.1M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (5m) 29s ago 16m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (6m) 2m ago 18m 508M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (5m) 2m ago 18m 47.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (4m) 29s ago 16m 39.1M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (5m) 2m ago 16m 38.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (6m) 2m ago 12m 18.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (6m) 29s ago 12m 18.6M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (4m) 2m ago 16m 51.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (3m) 2m ago 15m 52.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (2m) 2m ago 15m 40.8M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (2m) 2m ago 14m 14.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (104s) 29s ago 14m 57.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 2e371a2f9616 2024-05-06T12:42:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (69s) 29s ago 14m 53.1M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 13f5da2a25e3 2024-05-06T12:42:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (34s) 29s ago 13m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c803277a9f3d 2024-05-06T12:42:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (13m) 29s ago 13m 63.7M 2992M 17.2.0 e1d6a67b021e b98d9a542c75 2024-05-06T12:42:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (5m) 29s ago 12m 68.6M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:42:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (11m) 2m ago 11m 75.5M - 17.2.0 e1d6a67b021e e968ebdbe278 2024-05-06T12:42:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (11m) 29s ago 11m 73.9M - 17.2.0 e1d6a67b021e 270928ccc9a9 2024-05-06T12:42:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (10m) 2m ago 10m 75.1M - 17.2.0 e1d6a67b021e e5ee36d93a45 2024-05-06T12:42:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (10m) 29s ago 10m 73.8M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:42:47.510 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate[65327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate[65327]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 bash[65312]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 bash[65312]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 bash[65312]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 bash[65312]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 bash[65312]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 bash[65312]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 bash[65312]: --> ceph-volume raw activate successful for osd ID: 7 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate[65327]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate[65327]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate[65327]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate[65327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate[65327]: --> ceph-volume raw activate successful for osd ID: 7 2024-05-06T12:42:47.511 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 podman[65312]: 2024-05-06 12:42:47.198198967 +0000 UTC m=+0.826300665 container died 35ba4c1826a835a2328c848733a4370213dfd4518f8a3635e302256acbd6e55c (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 7 2024-05-06T12:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-06T12:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-05-06T12:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 12 2024-05-06T12:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:42:48.005 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:42:48.005 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:42:48.005 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:42:48.005 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:42:48.005 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:42:48.006 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:42:48.006 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-05-06T12:42:48.006 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:42:48.006 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "12/23 daemons upgraded", 2024-05-06T12:42:48.006 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T12:42:48.006 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:42:48.006 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:42:48.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:47 smithi086 podman[65460]: 2024-05-06 12:42:47.860016899 +0000 UTC m=+0.656562127 container remove 35ba4c1826a835a2328c848733a4370213dfd4518f8a3635e302256acbd6e55c (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-06T12:42:48.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:48 smithi086 podman[65522]: 2024-05-06 12:42:47.975323554 +0000 UTC m=+0.018222979 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:42:48.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:48 smithi086 podman[65522]: 2024-05-06 12:42:48.085753686 +0000 UTC m=+0.128653097 container create 6c537a696434276822568a543d233f650997770851ea4575ff6c75354de2b339 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-05-06T12:42:48.418 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid 2024-05-06T12:42:48.418 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] OSD_DOWN: 1 osds down 2024-05-06T12:42:48.418 INFO:teuthology.orchestra.run.smithi067.stdout: osd.7 (root=default,host=smithi086) is down 2024-05-06T12:42:48.418 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-05-06T12:42:48.418 INFO:teuthology.orchestra.run.smithi067.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-05-06T12:42:48.472 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:48 smithi086 ceph-mon[55252]: from='client.44461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:48 smithi086 ceph-mon[55252]: pgmap v204: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T12:42:48.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:48 smithi086 ceph-mon[55252]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:48 smithi086 ceph-mon[55252]: from='client.44473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.473 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:48 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1995323372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[64635]: from='client.44461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[64635]: pgmap v204: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T12:42:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[64635]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[64635]: from='client.44473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1995323372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[62070]: from='client.44461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[62070]: pgmap v204: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T12:42:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[62070]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[62070]: from='client.44473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:48.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:48 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1995323372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:48.869 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:48 smithi086 podman[65522]: 2024-05-06 12:42:48.472088297 +0000 UTC m=+0.514987710 container init 6c537a696434276822568a543d233f650997770851ea4575ff6c75354de2b339 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953) 2024-05-06T12:42:48.870 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:48 smithi086 podman[65522]: 2024-05-06 12:42:48.475219456 +0000 UTC m=+0.518118879 container start 6c537a696434276822568a543d233f650997770851ea4575ff6c75354de2b339 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-06T12:42:48.870 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:48 smithi086 ceph-osd[65541]: -- 172.21.15.86:0/1591358079 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55605b789680 con 0x55605b753000 2024-05-06T12:42:48.870 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:48 smithi086 bash[65522]: 6c537a696434276822568a543d233f650997770851ea4575ff6c75354de2b339 2024-05-06T12:42:48.870 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:48 smithi086 systemd[1]: Started Ceph osd.7 for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:42:49.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:49 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:42:49.017+0000 7fb62500d7c0 -1 Falling back to public interface 2024-05-06T12:42:49.666 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:49 smithi086 ceph-mon[55252]: from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:49.666 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:49 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1807681127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:42:49.666 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.666 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.666 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.667 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[62070]: from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1807681127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:42:49.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:49.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[64635]: from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:42:49.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1807681127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:42:49.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:49.727 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:42:50.696 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:50 smithi086 ceph-mon[55252]: pgmap v205: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:50.696 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:50 smithi086 ceph-mon[55252]: Health check failed: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:50 smithi067 ceph-mon[62070]: pgmap v205: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:50 smithi067 ceph-mon[62070]: Health check failed: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:50.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:50 smithi067 ceph-mon[64635]: pgmap v205: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:50.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:50 smithi067 ceph-mon[64635]: Health check failed: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded (PG_DEGRADED) 2024-05-06T12:42:51.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:51 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:51.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:51 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:51.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:51 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:42:52.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:42:51.748+0000 7fb62500d7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-05-06T12:42:52.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:51 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:42:51.779+0000 7fb62500d7c0 -1 osd.7 140 log_to_monitors true 2024-05-06T12:42:52.679 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:52 smithi086 ceph-mon[55252]: pgmap v206: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:52.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:52 smithi086 ceph-mon[55252]: from='osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:42:52.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:52 smithi086 ceph-mon[55252]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:42:52.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:52 smithi067 ceph-mon[62070]: pgmap v206: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:52.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:52 smithi067 ceph-mon[62070]: from='osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:42:52.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:52 smithi067 ceph-mon[62070]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:42:52.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:52 smithi067 ceph-mon[64635]: pgmap v206: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:52.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:52 smithi067 ceph-mon[64635]: from='osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:42:52.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:52 smithi067 ceph-mon[64635]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T12:42:53.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:42:52 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:42:52.918+0000 7fb61245c700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T12:42:53.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[64635]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T12:42:53.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[64635]: from='osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:53.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[64635]: osdmap e143: 8 total, 7 up, 8 in 2024-05-06T12:42:53.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[64635]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[64635]: from='osd.7 ' entity='osd.7' 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[62070]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[62070]: from='osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[62070]: osdmap e143: 8 total, 7 up, 8 in 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[62070]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[62070]: from='osd.7 ' entity='osd.7' 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:53.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:53 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:53.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:53 smithi086 ceph-mon[55252]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T12:42:53.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:53 smithi086 ceph-mon[55252]: from='osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:53.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:53 smithi086 ceph-mon[55252]: osdmap e143: 8 total, 7 up, 8 in 2024-05-06T12:42:53.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:53 smithi086 ceph-mon[55252]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-06T12:42:53.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:53 smithi086 ceph-mon[55252]: from='osd.7 ' entity='osd.7' 2024-05-06T12:42:53.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:53.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:53 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:54.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:42:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:42:53] "GET /metrics HTTP/1.1" 200 38613 "" "Prometheus/2.33.4" 2024-05-06T12:42:54.722 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:54 smithi086 ceph-mon[55252]: pgmap v208: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:54.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:54 smithi086 ceph-mon[55252]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:42:54.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:54 smithi086 ceph-mon[55252]: osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732] boot 2024-05-06T12:42:54.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:54 smithi086 ceph-mon[55252]: osdmap e144: 8 total, 8 up, 8 in 2024-05-06T12:42:54.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:54 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:42:54.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:54 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:54.723 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:54 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[62070]: pgmap v208: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[62070]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[62070]: osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732] boot 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[62070]: osdmap e144: 8 total, 8 up, 8 in 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[64635]: pgmap v208: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:54.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[64635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T12:42:54.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[64635]: osd.7 [v2:172.21.15.86:6824/1072138732,v1:172.21.15.86:6825/1072138732] boot 2024-05-06T12:42:54.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[64635]: osdmap e144: 8 total, 8 up, 8 in 2024-05-06T12:42:54.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T12:42:54.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:54.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:54 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:56.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:55 smithi086 ceph-mon[55252]: pgmap v210: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:56.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:55 smithi086 ceph-mon[55252]: osdmap e145: 8 total, 8 up, 8 in 2024-05-06T12:42:56.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:55 smithi067 ceph-mon[64635]: pgmap v210: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:56.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:55 smithi067 ceph-mon[64635]: osdmap e145: 8 total, 8 up, 8 in 2024-05-06T12:42:56.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:55 smithi067 ceph-mon[62070]: pgmap v210: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-05-06T12:42:56.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:55 smithi067 ceph-mon[62070]: osdmap e145: 8 total, 8 up, 8 in 2024-05-06T12:42:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: pgmap v212: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-05-06T12:42:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:42:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-06T12:42:58.227 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: pgmap v212: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:42:58.228 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-06T12:42:58.229 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-06T12:42:58.230 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-06T12:42:58.231 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-06T12:42:58.231 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:57 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: pgmap v212: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:42:58.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-06T12:42:58.371 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-06T12:42:58.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-06T12:42:58.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-06T12:42:58.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-06T12:42:58.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-06T12:42:58.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-06T12:42:58.372 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:57 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-06T12:42:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[62070]: Upgrade: Setting container_image for all osd 2024-05-06T12:42:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[62070]: Upgrade: Setting require_osd_release to 19 squid 2024-05-06T12:42:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[62070]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-06T12:42:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-06T12:42:59.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[62070]: osdmap e146: 8 total, 8 up, 8 in 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[64635]: Upgrade: Setting container_image for all osd 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[64635]: Upgrade: Setting require_osd_release to 19 squid 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[64635]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[64635]: osdmap e146: 8 total, 8 up, 8 in 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:59.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:58 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:42:59.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:58 smithi086 ceph-mon[55252]: Upgrade: Setting container_image for all osd 2024-05-06T12:42:59.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:58 smithi086 ceph-mon[55252]: Upgrade: Setting require_osd_release to 19 squid 2024-05-06T12:42:59.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:58 smithi086 ceph-mon[55252]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-06T12:42:59.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:58 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-06T12:42:59.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:58 smithi086 ceph-mon[55252]: osdmap e146: 8 total, 8 up, 8 in 2024-05-06T12:42:59.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:58 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:42:59.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:58 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[62070]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-06T12:43:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[62070]: Upgrade: Setting container_image for all mds 2024-05-06T12:43:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[62070]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded) 2024-05-06T12:43:00.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[62070]: Cluster is now healthy 2024-05-06T12:43:00.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[64635]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-06T12:43:00.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[64635]: Upgrade: Setting container_image for all mds 2024-05-06T12:43:00.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[64635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded) 2024-05-06T12:43:00.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:42:59 smithi067 ceph-mon[64635]: Cluster is now healthy 2024-05-06T12:43:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:59 smithi086 ceph-mon[55252]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-06T12:43:00.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:59 smithi086 ceph-mon[55252]: Upgrade: Setting container_image for all mds 2024-05-06T12:43:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:59 smithi086 ceph-mon[55252]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded) 2024-05-06T12:43:00.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:42:59 smithi086 ceph-mon[55252]: Cluster is now healthy 2024-05-06T12:43:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:01.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:01.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:01.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:00 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:01.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:00 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:00 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:00 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi067.iyetmn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:01.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:00 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:02.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:01 smithi067 ceph-mon[62070]: Upgrade: Updating rgw.foo.smithi067.iyetmn (1/4) 2024-05-06T12:43:02.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:01 smithi067 ceph-mon[62070]: Deploying daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:43:02.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:01 smithi067 ceph-mon[62070]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-05-06T12:43:02.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:01 smithi067 ceph-mon[64635]: Upgrade: Updating rgw.foo.smithi067.iyetmn (1/4) 2024-05-06T12:43:02.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:01 smithi067 ceph-mon[64635]: Deploying daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:43:02.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:01 smithi067 ceph-mon[64635]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-05-06T12:43:02.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:01 smithi086 ceph-mon[55252]: Upgrade: Updating rgw.foo.smithi067.iyetmn (1/4) 2024-05-06T12:43:02.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:01 smithi086 ceph-mon[55252]: Deploying daemon rgw.foo.smithi067.iyetmn on smithi067 2024-05-06T12:43:02.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:01 smithi086 ceph-mon[55252]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-05-06T12:43:03.932 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:43:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:43:03] "GET /metrics HTTP/1.1" 200 38613 "" "Prometheus/2.33.4" 2024-05-06T12:43:04.599 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[64635]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:43:04.599 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:04.599 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:04.599 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:04.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[62070]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:43:04.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:04.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:04.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:04.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:04 smithi086 ceph-mon[55252]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:43:04.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:04.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:04.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:05 smithi086 ceph-mon[55252]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:43:06.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:06.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:05 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:06.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:05 smithi067 ceph-mon[62070]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:43:06.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:06.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:05 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:06.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:05 smithi067 ceph-mon[64635]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T12:43:06.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:06.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:05 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:07.719 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[62070]: Upgrade: Updating rgw.smpl.smithi067.wpnjcz (2/4) 2024-05-06T12:43:07.719 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:07.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:07.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:07.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[62070]: Deploying daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:43:07.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[62070]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:43:07.721 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[64635]: Upgrade: Updating rgw.smpl.smithi067.wpnjcz (2/4) 2024-05-06T12:43:07.721 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:07.721 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:07.721 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:07.721 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[64635]: Deploying daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:43:07.721 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:07 smithi067 ceph-mon[64635]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:43:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:07 smithi086 ceph-mon[55252]: Upgrade: Updating rgw.smpl.smithi067.wpnjcz (2/4) 2024-05-06T12:43:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi067.wpnjcz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:07 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:07 smithi086 ceph-mon[55252]: Deploying daemon rgw.smpl.smithi067.wpnjcz on smithi067 2024-05-06T12:43:07.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:07 smithi086 ceph-mon[55252]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:43:10.117 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:09 smithi067 ceph-mon[62070]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 102 B/s wr, 101 op/s 2024-05-06T12:43:10.117 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:09 smithi067 ceph-mon[64635]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 102 B/s wr, 101 op/s 2024-05-06T12:43:10.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:09 smithi086 ceph-mon[55252]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 102 B/s wr, 101 op/s 2024-05-06T12:43:11.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:10 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:11.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:10 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:11.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:10 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:10 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:10 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:11.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:10 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:11.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:10 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:11.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:10 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:11.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:10 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:11.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:11 smithi086 ceph-mon[55252]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 86 B/s wr, 85 op/s 2024-05-06T12:43:12.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:11 smithi067 ceph-mon[62070]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 86 B/s wr, 85 op/s 2024-05-06T12:43:12.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:11 smithi067 ceph-mon[64635]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 86 B/s wr, 85 op/s 2024-05-06T12:43:13.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:13 smithi086 ceph-mon[55252]: Upgrade: Updating rgw.foo.smithi086.frwvbk (3/4) 2024-05-06T12:43:13.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:13.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:13.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:13 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:13.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:13 smithi086 ceph-mon[55252]: Deploying daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:43:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[62070]: Upgrade: Updating rgw.foo.smithi086.frwvbk (3/4) 2024-05-06T12:43:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:13.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[62070]: Deploying daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:43:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[64635]: Upgrade: Updating rgw.foo.smithi086.frwvbk (3/4) 2024-05-06T12:43:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi086.frwvbk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:13.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:13 smithi067 ceph-mon[64635]: Deploying daemon rgw.foo.smithi086.frwvbk on smithi086 2024-05-06T12:43:14.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:43:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:43:13] "GET /metrics HTTP/1.1" 200 38616 "" "Prometheus/2.33.4" 2024-05-06T12:43:14.439 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:14 smithi086 ceph-mon[55252]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 85 B/s wr, 154 op/s 2024-05-06T12:43:14.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:14 smithi067 ceph-mon[62070]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 85 B/s wr, 154 op/s 2024-05-06T12:43:14.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:14 smithi067 ceph-mon[64635]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 85 B/s wr, 154 op/s 2024-05-06T12:43:16.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:16 smithi086 ceph-mon[55252]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:16.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:16 smithi067 ceph-mon[62070]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:16.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:16 smithi067 ceph-mon[64635]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:17.787 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:17 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:17.787 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:17 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:17.787 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:17 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:17.787 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:17 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:17 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:17 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:18.783 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:18 smithi086 ceph-mon[55252]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:18.784 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:18.784 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:18.784 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:18 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:18.801 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[62070]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:18.801 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:18.802 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:18.802 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:18.802 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[64635]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:18.802 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:18.802 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi086.myvasn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T12:43:18.802 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:18 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:18.803 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:43:19.479 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:43:19.479 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (7m) 2m ago 12m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:43:19.479 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (7m) 26s ago 12m 50.0M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:43:19.479 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 running (6m) 2m ago 11m 53.1M - 3.5 e1d6a67b021e 69e7abad2b84 2024-05-06T12:43:19.479 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (6m) 26s ago 16m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (7m) 2m ago 18m 508M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (6m) 2m ago 18m 47.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (5m) 26s ago 17m 39.7M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (5m) 2m ago 17m 38.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (7m) 2m ago 13m 18.5M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (7m) 26s ago 13m 15.7M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (4m) 2m ago 16m 51.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (4m) 2m ago 16m 52.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (3m) 2m ago 15m 40.8M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (2m) 2m ago 15m 14.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (2m) 26s ago 15m 58.1M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 2e371a2f9616 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (101s) 26s ago 14m 54.7M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 13f5da2a25e3 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (66s) 26s ago 14m 54.3M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c803277a9f3d 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (31s) 26s ago 13m 12.7M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6c537a696434 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (6m) 26s ago 13m 70.7M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:43:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 starting - - - - 2024-05-06T12:43:19.481 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 starting - - - - 2024-05-06T12:43:19.481 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 starting - - - - 2024-05-06T12:43:19.481 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (11m) 26s ago 11m 74.0M - 17.2.0 e1d6a67b021e 3f278c2006b6 2024-05-06T12:43:19.739 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:19 smithi086 ceph-mon[55252]: Upgrade: Updating rgw.smpl.smithi086.myvasn (4/4) 2024-05-06T12:43:19.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:19 smithi086 ceph-mon[55252]: Deploying daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:43:19.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:19.894 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:43:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-06T12:43:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 16 2024-05-06T12:43:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:43:19.896 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:43:19.896 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:19 smithi067 ceph-mon[62070]: Upgrade: Updating rgw.smpl.smithi086.myvasn (4/4) 2024-05-06T12:43:19.896 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:19 smithi067 ceph-mon[62070]: Deploying daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:43:19.896 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:19.896 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:19 smithi067 ceph-mon[64635]: Upgrade: Updating rgw.smpl.smithi086.myvasn (4/4) 2024-05-06T12:43:19.896 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:19 smithi067 ceph-mon[64635]: Deploying daemon rgw.smpl.smithi086.myvasn on smithi086 2024-05-06T12:43:19.897 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-05-06T12:43:20.238 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:43:20.239 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "13/23 daemons upgraded", 2024-05-06T12:43:20.239 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading rgw daemons", 2024-05-06T12:43:20.239 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-05-06T12:43:20.239 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:43:20.655 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_OK 2024-05-06T12:43:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:20 smithi086 ceph-mon[55252]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-05-06T12:43:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:20 smithi086 ceph-mon[55252]: from='client.44560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:20 smithi086 ceph-mon[55252]: from='client.44566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:20 smithi086 ceph-mon[55252]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:20 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1568691870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:20 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:20.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:20 smithi086 ceph-mon[55252]: from='client.44584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[64635]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-05-06T12:43:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[64635]: from='client.44560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[64635]: from='client.44566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[64635]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1568691870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[64635]: from='client.44584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[62070]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-05-06T12:43:20.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[62070]: from='client.44560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[62070]: from='client.44566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[62070]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:20.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1568691870' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:20.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:20.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:20 smithi067 ceph-mon[62070]: from='client.44584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:21.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:21 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/4220376970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:43:21.844 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:21 smithi086 ceph-mon[55252]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:21.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:21 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/4220376970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:43:21.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:21 smithi067 ceph-mon[62070]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:21.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:21 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/4220376970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:43:21.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:21 smithi067 ceph-mon[64635]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:23.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:23.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:23 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:23.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:23.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:23 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:23.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:23 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:24.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:43:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:43:23] "GET /metrics HTTP/1.1" 200 38617 "" "Prometheus/2.33.4" 2024-05-06T12:43:24.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:24 smithi086 ceph-mon[55252]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 175 op/s 2024-05-06T12:43:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:24 smithi067 ceph-mon[62070]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 175 op/s 2024-05-06T12:43:24.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:24 smithi067 ceph-mon[64635]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 175 op/s 2024-05-06T12:43:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[64635]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 255 B/s wr, 118 op/s 2024-05-06T12:43:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[62070]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 255 B/s wr, 118 op/s 2024-05-06T12:43:26.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:26 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:26 smithi086 ceph-mon[55252]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 255 B/s wr, 118 op/s 2024-05-06T12:43:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:26 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:26 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:26 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:26 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[62070]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 170 B/s wr, 104 op/s 2024-05-06T12:43:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[64635]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 170 B/s wr, 104 op/s 2024-05-06T12:43:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:28 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:28 smithi086 ceph-mon[55252]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 170 B/s wr, 104 op/s 2024-05-06T12:43:28.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:28.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:28 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:30.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:30 smithi086 ceph-mon[55252]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:30.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:30 smithi067 ceph-mon[62070]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:30.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:30 smithi067 ceph-mon[64635]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:31.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:31.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:31 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:31.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:31.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:31.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:31.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:31 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: Detected new or changed devices on smithi086 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: Detected new or changed devices on smithi067 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: Upgrade: Setting container_image for all rgw 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi067.iyetmn"}]: dispatch 2024-05-06T12:43:32.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi067.iyetmn"}]': finished 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.frwvbk"}]: dispatch 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.frwvbk"}]': finished 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi067.wpnjcz"}]: dispatch 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi067.wpnjcz"}]': finished 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.myvasn"}]: dispatch 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.myvasn"}]': finished 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T12:43:32.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:32 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T12:43:32.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: Detected new or changed devices on smithi086 2024-05-06T12:43:32.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: Detected new or changed devices on smithi067 2024-05-06T12:43:32.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:43:32.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: Upgrade: Setting container_image for all rgw 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi067.iyetmn"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi067.iyetmn"}]': finished 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.frwvbk"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.frwvbk"}]': finished 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi067.wpnjcz"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi067.wpnjcz"}]': finished 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.myvasn"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.myvasn"}]': finished 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.727 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: Detected new or changed devices on smithi086 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: Detected new or changed devices on smithi067 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.728 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: Upgrade: Setting container_image for all rgw 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi067.iyetmn"}]: dispatch 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi067.iyetmn"}]': finished 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.frwvbk"}]: dispatch 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi086.frwvbk"}]': finished 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi067.wpnjcz"}]: dispatch 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi067.wpnjcz"}]': finished 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.myvasn"}]: dispatch 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi086.myvasn"}]': finished 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:32.729 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:32.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T12:43:32.730 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:32 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:33.777 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:43:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:43:33] "GET /metrics HTTP/1.1" 200 38617 "" "Prometheus/2.33.4" 2024-05-06T12:43:34.222 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[64635]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T12:43:34.222 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[64635]: Upgrade: Updating iscsi.foo.smithi067.fnpkny 2024-05-06T12:43:34.222 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[64635]: Deploying daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[62070]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[62070]: Upgrade: Updating iscsi.foo.smithi067.fnpkny 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[62070]: Deploying daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:43:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:33 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:33 smithi086 ceph-mon[55252]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T12:43:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:33 smithi086 ceph-mon[55252]: Upgrade: Updating iscsi.foo.smithi067.fnpkny 2024-05-06T12:43:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi067.fnpkny", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T12:43:34.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:34.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:33 smithi086 ceph-mon[55252]: Deploying daemon iscsi.foo.smithi067.fnpkny on smithi067 2024-05-06T12:43:34.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:33 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:36.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:35 smithi067 ceph-mon[62070]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 85 B/s wr, 77 op/s 2024-05-06T12:43:36.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:35 smithi067 ceph-mon[64635]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 85 B/s wr, 77 op/s 2024-05-06T12:43:36.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:35 smithi086 ceph-mon[55252]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 85 B/s wr, 77 op/s 2024-05-06T12:43:38.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:37 smithi067 ceph-mon[62070]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 63 op/s 2024-05-06T12:43:38.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:37 smithi067 ceph-mon[64635]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 63 op/s 2024-05-06T12:43:38.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:37 smithi086 ceph-mon[55252]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 63 op/s 2024-05-06T12:43:40.171 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:39 smithi086 ceph-mon[55252]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 64 op/s 2024-05-06T12:43:40.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:39 smithi067 ceph-mon[62070]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 64 op/s 2024-05-06T12:43:40.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:39 smithi067 ceph-mon[64635]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 64 op/s 2024-05-06T12:43:41.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:40 smithi067 ceph-mon[62070]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:41.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:40 smithi067 ceph-mon[64635]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:41.316 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:40 smithi086 ceph-mon[55252]: from='client.25172 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:43:42.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:41 smithi067 ceph-mon[62070]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:43:42.225 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:41 smithi067 ceph-mon[64635]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:43:42.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:41 smithi086 ceph-mon[55252]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:43:44.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:43:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:43:43] "GET /metrics HTTP/1.1" 200 38684 "" "Prometheus/2.33.4" 2024-05-06T12:43:44.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:43 smithi067 ceph-mon[62070]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:43:44.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:43 smithi067 ceph-mon[64635]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:43:44.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:43 smithi086 ceph-mon[55252]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:43:46.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:45 smithi067 ceph-mon[62070]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:43:46.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:45 smithi067 ceph-mon[64635]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:43:46.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:45 smithi086 ceph-mon[55252]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:43:46.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:43:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:43:46.609+0000 7f333b5d2700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. 2024-05-06T12:43:47.658 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:47 smithi067 ceph-mon[62070]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. 2024-05-06T12:43:47.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:47 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:47.659 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:47 smithi067 ceph-mon[62070]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-05-06T12:43:47.659 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:47 smithi067 ceph-mon[64635]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. 2024-05-06T12:43:47.659 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:47 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:47.659 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:47 smithi067 ceph-mon[64635]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-05-06T12:43:47.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:47 smithi086 ceph-mon[55252]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. 2024-05-06T12:43:47.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:47 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:47.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:47 smithi086 ceph-mon[55252]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-05-06T12:43:48.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:48 smithi067 ceph-mon[64635]: Health check failed: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-06T12:43:48.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:48 smithi067 ceph-mon[62070]: Health check failed: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-06T12:43:49.034 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:48 smithi086 ceph-mon[55252]: Health check failed: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-06T12:43:49.999 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:49 smithi067 ceph-mon[62070]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:43:50.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:50.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:50.000 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:49 smithi067 ceph-mon[64635]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:43:50.000 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:50.000 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:49 smithi086 ceph-mon[55252]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:43:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:50.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:43:51.053 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:43:51.410 INFO:teuthology.orchestra.run.smithi067.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed." 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:51.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:51.412 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:51 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:43:51.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:51 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:43:52.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T12:43:52.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-05-06T12:43:52.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-05-06T12:43:52.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: Checking dashboard <-> RGW credentials 2024-05-06T12:43:52.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: from='client.54533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:52.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 370 B/s rd, 0 op/s 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 489 B/s rd, 0 op/s 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[64635]: from='client.44656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: Checking dashboard <-> RGW credentials 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: from='client.54533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 370 B/s rd, 0 op/s 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 489 B/s rd, 0 op/s 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:43:52.476 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:52 smithi067 ceph-mon[62070]: from='client.44656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: Checking dashboard <-> RGW credentials 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: from='client.54533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 370 B/s rd, 0 op/s 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 489 B/s rd, 0 op/s 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:43:52.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:52 smithi086 ceph-mon[55252]: from='client.44656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:53.975 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:43:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:43:53] "GET /metrics HTTP/1.1" 200 38760 "" "Prometheus/2.33.4" 2024-05-06T12:43:54.141 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (7m) 3s ago 12m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (7m) 3s ago 12m 50.5M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 unknown 3s ago 11m - - 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (7m) 3s ago 17m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (7m) 3s ago 19m 514M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (6m) 3s ago 19m 56.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (6m) 3s ago 17m 41.4M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (6m) 3s ago 17m 47.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (8m) 3s ago 13m 19.3M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (7m) 3s ago 13m 15.5M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:43:54.142 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (5m) 3s ago 17m 62.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (4m) 3s ago 16m 65.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (4m) 3s ago 16m 49.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (3m) 3s ago 16m 54.5M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (2m) 3s ago 15m 63.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 2e371a2f9616 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (2m) 3s ago 15m 59.4M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 13f5da2a25e3 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (101s) 3s ago 14m 57.4M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c803277a9f3d 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (66s) 3s ago 14m 57.3M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6c537a696434 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (7m) 3s ago 13m 70.7M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (49s) 3s ago 12m 75.5M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 33e08e639a62 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (38s) 3s ago 12m 77.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c7074d9a3456 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (43s) 3s ago 12m 75.3M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5fd5e7ec35e7 2024-05-06T12:43:54.143 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (32s) 3s ago 12m 77.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 54cb1e78b787 2024-05-06T12:43:54.468 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:54 smithi067 ceph-mon[62070]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 4 op/s 2024-05-06T12:43:54.468 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:54 smithi067 ceph-mon[64635]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 4 op/s 2024-05-06T12:43:54.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:54 smithi086 ceph-mon[55252]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 4 op/s 2024-05-06T12:43:54.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T12:43:55.448 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:55 smithi067 ceph-mon[64635]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:55.448 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:55 smithi067 ceph-mon[62070]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:55.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:55 smithi086 ceph-mon[55252]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:43:56.396 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:56 smithi067 ceph-mon[62070]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-05-06T12:43:56.396 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:56 smithi067 ceph-mon[64635]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-05-06T12:43:56.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:56 smithi086 ceph-mon[55252]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-05-06T12:43:56.934 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:43:56.934 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:43:56.934 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:43:56.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 4 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 17 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:43:56.935 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:43:57.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:57 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/393961303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:57.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:57 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/393961303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:57.699 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:57 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/393961303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:43:58.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-05-06T12:43:58.577 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:58 smithi067 ceph-mon[62070]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-05-06T12:43:58.577 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:58 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1313819767' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:43:58.578 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:58 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/993079833' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2029960657"}]: dispatch 2024-05-06T12:43:58.578 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:58 smithi067 ceph-mon[64635]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-05-06T12:43:58.578 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:58 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1313819767' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:43:58.578 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:58 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/993079833' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2029960657"}]: dispatch 2024-05-06T12:43:58.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:58 smithi086 ceph-mon[55252]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-05-06T12:43:58.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:58 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1313819767' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T12:43:58.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:58 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/993079833' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2029960657"}]: dispatch 2024-05-06T12:43:59.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:59 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/993079833' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2029960657"}]': finished 2024-05-06T12:43:59.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:59 smithi086 ceph-mon[55252]: osdmap e147: 8 total, 8 up, 8 in 2024-05-06T12:43:59.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:43:59 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1916264352' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1103954612"}]: dispatch 2024-05-06T12:43:59.637 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:59 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/993079833' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2029960657"}]': finished 2024-05-06T12:43:59.637 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:59 smithi067 ceph-mon[62070]: osdmap e147: 8 total, 8 up, 8 in 2024-05-06T12:43:59.637 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:43:59 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1916264352' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1103954612"}]: dispatch 2024-05-06T12:43:59.637 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:59 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/993079833' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/2029960657"}]': finished 2024-05-06T12:43:59.637 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:59 smithi067 ceph-mon[64635]: osdmap e147: 8 total, 8 up, 8 in 2024-05-06T12:43:59.637 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:43:59 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1916264352' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1103954612"}]: dispatch 2024-05-06T12:43:59.869 INFO:teuthology.orchestra.run.smithi067.stdout:wait for servicemap items w/ changing names to refresh 2024-05-06T12:44:00.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 60' 2024-05-06T12:44:00.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:00 smithi086 ceph-mon[55252]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-05-06T12:44:00.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:00 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/1916264352' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1103954612"}]': finished 2024-05-06T12:44:00.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:00 smithi086 ceph-mon[55252]: osdmap e148: 8 total, 8 up, 8 in 2024-05-06T12:44:00.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:00 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3805654688' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1408603586"}]: dispatch 2024-05-06T12:44:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[62070]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-05-06T12:44:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/1916264352' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1103954612"}]': finished 2024-05-06T12:44:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[62070]: osdmap e148: 8 total, 8 up, 8 in 2024-05-06T12:44:00.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3805654688' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1408603586"}]: dispatch 2024-05-06T12:44:00.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[64635]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-05-06T12:44:00.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/1916264352' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6800/1103954612"}]': finished 2024-05-06T12:44:00.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[64635]: osdmap e148: 8 total, 8 up, 8 in 2024-05-06T12:44:00.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:00 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3805654688' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1408603586"}]: dispatch 2024-05-06T12:44:01.360 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3805654688' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1408603586"}]': finished 2024-05-06T12:44:01.361 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[64635]: osdmap e149: 8 total, 8 up, 8 in 2024-05-06T12:44:01.361 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3095946730' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]: dispatch 2024-05-06T12:44:01.361 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[64635]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]: dispatch 2024-05-06T12:44:01.362 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3805654688' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1408603586"}]': finished 2024-05-06T12:44:01.362 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[62070]: osdmap e149: 8 total, 8 up, 8 in 2024-05-06T12:44:01.362 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3095946730' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]: dispatch 2024-05-06T12:44:01.362 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:01 smithi067 ceph-mon[62070]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]: dispatch 2024-05-06T12:44:01.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:01 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3805654688' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1408603586"}]': finished 2024-05-06T12:44:01.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:01 smithi086 ceph-mon[55252]: osdmap e149: 8 total, 8 up, 8 in 2024-05-06T12:44:01.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:01 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3095946730' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]: dispatch 2024-05-06T12:44:01.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:01 smithi086 ceph-mon[55252]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]: dispatch 2024-05-06T12:44:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[62070]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T12:44:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[62070]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]': finished 2024-05-06T12:44:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[62070]: osdmap e150: 8 total, 8 up, 8 in 2024-05-06T12:44:02.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/2399565702' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1103954612"}]: dispatch 2024-05-06T12:44:02.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[64635]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T12:44:02.475 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[64635]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]': finished 2024-05-06T12:44:02.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[64635]: osdmap e150: 8 total, 8 up, 8 in 2024-05-06T12:44:02.476 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:02 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/2399565702' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1103954612"}]: dispatch 2024-05-06T12:44:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:02 smithi086 ceph-mon[55252]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T12:44:02.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:02 smithi086 ceph-mon[55252]: from='client.? ' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/586064548"}]': finished 2024-05-06T12:44:02.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:02 smithi086 ceph-mon[55252]: osdmap e150: 8 total, 8 up, 8 in 2024-05-06T12:44:02.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:02 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/2399565702' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1103954612"}]: dispatch 2024-05-06T12:44:03.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:03 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/2399565702' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1103954612"}]': finished 2024-05-06T12:44:03.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:03 smithi086 ceph-mon[55252]: osdmap e151: 8 total, 8 up, 8 in 2024-05-06T12:44:03.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:03 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3218141644' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1915611504"}]: dispatch 2024-05-06T12:44:03.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:03 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/2399565702' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1103954612"}]': finished 2024-05-06T12:44:03.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:03 smithi067 ceph-mon[64635]: osdmap e151: 8 total, 8 up, 8 in 2024-05-06T12:44:03.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:03 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3218141644' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1915611504"}]: dispatch 2024-05-06T12:44:03.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:03 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/2399565702' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:6801/1103954612"}]': finished 2024-05-06T12:44:03.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:03 smithi067 ceph-mon[62070]: osdmap e151: 8 total, 8 up, 8 in 2024-05-06T12:44:03.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:03 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3218141644' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1915611504"}]: dispatch 2024-05-06T12:44:04.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:44:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:44:03] "GET /metrics HTTP/1.1" 200 38760 "" "Prometheus/2.33.4" 2024-05-06T12:44:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:04 smithi086 ceph-mon[55252]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:04 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3218141644' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1915611504"}]': finished 2024-05-06T12:44:04.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:04 smithi086 ceph-mon[55252]: osdmap e152: 8 total, 8 up, 8 in 2024-05-06T12:44:04.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:44:04.620 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[64635]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3218141644' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1915611504"}]': finished 2024-05-06T12:44:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[64635]: osdmap e152: 8 total, 8 up, 8 in 2024-05-06T12:44:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:44:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:04.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[62070]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:04.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3218141644' entity='client.iscsi.foo.smithi067.fnpkny' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.67:0/1915611504"}]': finished 2024-05-06T12:44:04.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[62070]: osdmap e152: 8 total, 8 up, 8 in 2024-05-06T12:44:04.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:44:04.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:06.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:06 smithi086 ceph-mon[55252]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:06.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:06 smithi067 ceph-mon[62070]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:06.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:06 smithi067 ceph-mon[64635]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:08 smithi086 ceph-mon[55252]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:08.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:08 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:08 smithi067 ceph-mon[64635]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:08.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:08 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:08 smithi067 ceph-mon[62070]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:08.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:08 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:10.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:10 smithi086 ceph-mon[55252]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:44:10.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:10 smithi067 ceph-mon[64635]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:44:10.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:10 smithi067 ceph-mon[62070]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T12:44:12.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:12 smithi067 ceph-mon[64635]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:12.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:12 smithi067 ceph-mon[62070]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:12 smithi086 ceph-mon[55252]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:14.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:44:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:44:13] "GET /metrics HTTP/1.1" 200 38757 "" "Prometheus/2.33.4" 2024-05-06T12:44:14.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:14 smithi067 ceph-mon[62070]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:44:14.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:14 smithi067 ceph-mon[64635]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:44:14.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:14 smithi086 ceph-mon[55252]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:44:16.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:16 smithi067 ceph-mon[62070]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-05-06T12:44:16.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:16 smithi067 ceph-mon[64635]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-05-06T12:44:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:16 smithi086 ceph-mon[55252]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-05-06T12:44:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:18 smithi067 ceph-mon[62070]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:18 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:18.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:18 smithi067 ceph-mon[64635]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:18.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:18 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:18.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:18 smithi086 ceph-mon[55252]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:18.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:18 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:19.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:19.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:20.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:20 smithi067 ceph-mon[62070]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:20.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:20 smithi067 ceph-mon[64635]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:20 smithi086 ceph-mon[55252]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:22.390 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:22 smithi067 ceph-mon[62070]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:22.391 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:22 smithi067 ceph-mon[64635]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:22.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:22 smithi086 ceph-mon[55252]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:24.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:44:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:44:23] "GET /metrics HTTP/1.1" 200 38755 "" "Prometheus/2.33.4" 2024-05-06T12:44:24.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:24 smithi067 ceph-mon[64635]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:24 smithi067 ceph-mon[62070]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:24.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:24 smithi086 ceph-mon[55252]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:26.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:26 smithi067 ceph-mon[62070]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:26.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:26 smithi067 ceph-mon[64635]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:26.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:26 smithi086 ceph-mon[55252]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:28.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:28 smithi067 ceph-mon[64635]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:28.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:28 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:28.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:28 smithi067 ceph-mon[62070]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:28.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:28 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:28.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:28 smithi086 ceph-mon[55252]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:28.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:28 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:30.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:30 smithi067 ceph-mon[62070]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:30.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:30 smithi067 ceph-mon[64635]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:30.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:30 smithi086 ceph-mon[55252]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:32.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:32 smithi067 ceph-mon[64635]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:32.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:32 smithi067 ceph-mon[62070]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:32.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:32 smithi086 ceph-mon[55252]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:34.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:44:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:44:33] "GET /metrics HTTP/1.1" 200 38755 "" "Prometheus/2.33.4" 2024-05-06T12:44:34.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:34 smithi067 ceph-mon[62070]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:34.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:34 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:34.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:34 smithi067 ceph-mon[64635]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:34.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:34 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:34 smithi086 ceph-mon[55252]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:34.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:34 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:36.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:36 smithi067 ceph-mon[62070]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:36.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:36 smithi067 ceph-mon[64635]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:36.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:36 smithi086 ceph-mon[55252]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:38 smithi067 ceph-mon[62070]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:38.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:38 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:38.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:38 smithi067 ceph-mon[64635]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:38.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:38 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:38.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:38 smithi086 ceph-mon[55252]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:38.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:38 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:40.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:40 smithi067 ceph-mon[62070]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:40.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:40 smithi067 ceph-mon[64635]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:40.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:40 smithi086 ceph-mon[55252]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:42.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:42 smithi067 ceph-mon[64635]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:42.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:42 smithi067 ceph-mon[62070]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:42.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:42 smithi086 ceph-mon[55252]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T12:44:44.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:44:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:44:43] "GET /metrics HTTP/1.1" 200 38760 "" "Prometheus/2.33.4" 2024-05-06T12:44:44.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:44 smithi067 ceph-mon[62070]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:44.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:44 smithi067 ceph-mon[64635]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:44.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:44 smithi086 ceph-mon[55252]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:46.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:46 smithi067 ceph-mon[62070]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:46.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:46 smithi067 ceph-mon[64635]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:46.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:46 smithi086 ceph-mon[55252]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:48 smithi067 ceph-mon[62070]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:48.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:48 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:48.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:48 smithi067 ceph-mon[64635]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:48.726 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:48 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:48.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:48 smithi086 ceph-mon[55252]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:48.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:48 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:49.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:49 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:49.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:49 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:49.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:49 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:44:50.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:50 smithi067 ceph-mon[64635]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:50.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:50 smithi067 ceph-mon[62070]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:50.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:50 smithi086 ceph-mon[55252]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:44:52.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:52 smithi086 ceph-mon[55252]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:52.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:44:52.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:44:52.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:52 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:44:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[62070]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:44:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:44:52.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:44:52.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[64635]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:44:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T12:44:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T12:44:52.976 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:52 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' 2024-05-06T12:44:53.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:53 smithi067 ceph-mon[62070]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-05-06T12:44:53.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:53 smithi067 ceph-mon[62070]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-05-06T12:44:53.743 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:53 smithi067 ceph-mon[64635]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-05-06T12:44:53.744 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:53 smithi067 ceph-mon[64635]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-05-06T12:44:53.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:53 smithi086 ceph-mon[55252]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-05-06T12:44:53.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:53 smithi086 ceph-mon[55252]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-05-06T12:44:54.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:44:53 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:44:53] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-05-06T12:44:55.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:55 smithi086 ceph-mon[55252]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:55.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:55 smithi067 ceph-mon[62070]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:55.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:55 smithi067 ceph-mon[64635]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:57.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:57 smithi086 ceph-mon[55252]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:57.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:57 smithi067 ceph-mon[62070]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:57.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:57 smithi067 ceph-mon[64635]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:58.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:58 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:58.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:58 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:58.876 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:58 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:44:59.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:44:59 smithi086 ceph-mon[55252]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:59.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:44:59 smithi067 ceph-mon[62070]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:44:59.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:44:59 smithi067 ceph-mon[64635]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:45:01.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:01 smithi086 ceph-mon[55252]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:45:01.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:01 smithi067 ceph-mon[62070]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:45:01.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:01 smithi067 ceph-mon[64635]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T12:45:03.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T12:45:03.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:03 smithi067 ceph-mon[62070]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:45:03.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:03 smithi067 ceph-mon[64635]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:45:03.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:03 smithi086 ceph-mon[55252]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T12:45:04.154 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:03 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:45:03] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-05-06T12:45:04.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:04 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:45:04.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:04 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:45:04.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:04 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.a smithi067 *:9093,9094 running (9m) 74s ago 14m 23.5M - 0.23.0 ba2b418f427c 130b40fe1256 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.a smithi086 *:3000 running (8m) 74s ago 13m 50.5M - 8.3.5 dad864ee21e9 3a4e37b2de3a 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:iscsi.foo.smithi067.fnpkny smithi067 unknown 74s ago 13m - - 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi086 *:8443,9283,8765 running (8m) 74s ago 18m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 cad655065a4e 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi067 *:8443,9283,8765 running (8m) 74s ago 20m 514M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c0f8319f1bf4 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (7m) 74s ago 20m 56.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 60a77a61933a 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi086 running (7m) 74s ago 18m 41.4M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 3215d78b6d70 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi067 running (7m) 74s ago 19m 47.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 120694fd2a6c 2024-05-06T12:45:05.236 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.a smithi067 *:9100 running (9m) 74s ago 14m 19.3M - 1.3.1 1dbe0e931976 347a17f8d7d6 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.b smithi086 *:9100 running (8m) 74s ago 14m 15.5M - 1.3.1 1dbe0e931976 f4564f85d023 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (6m) 74s ago 18m 62.3M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 e278b8d0f47f 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (5m) 74s ago 18m 65.4M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b2661afeacd0 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (5m) 74s ago 17m 49.0M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 b6fce9672451 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (4m) 74s ago 17m 54.5M 2736M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7ec6d2a3db73 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi086 running (4m) 74s ago 16m 63.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 2e371a2f9616 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi086 running (3m) 74s ago 16m 59.4M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 13f5da2a25e3 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi086 running (2m) 74s ago 16m 57.4M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c803277a9f3d 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi086 running (2m) 74s ago 15m 57.3M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6c537a696434 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.a smithi086 *:9095 running (8m) 74s ago 15m 70.7M - 2.33.4 514e6a882f6e 8dffac3aef3b 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi067.iyetmn smithi067 *:8000 running (2m) 74s ago 13m 75.5M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 33e08e639a62 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.foo.smithi086.frwvbk smithi086 *:8000 running (109s) 74s ago 13m 77.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 c7074d9a3456 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi067.wpnjcz smithi067 *:80 running (114s) 74s ago 13m 75.3M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5fd5e7ec35e7 2024-05-06T12:45:05.237 INFO:teuthology.orchestra.run.smithi067.stdout:rgw.smpl.smithi086.myvasn smithi086 *:80 running (103s) 74s ago 13m 77.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 54cb1e78b787 2024-05-06T12:45:05.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:05 smithi067 ceph-mon[62070]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:05.839 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:05 smithi067 ceph-mon[64635]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:05.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T12:45:05.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:05 smithi086 ceph-mon[55252]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:06.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:06 smithi067 ceph-mon[62070]: from='client.44734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:06.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:06 smithi067 ceph-mon[64635]: from='client.44734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:06.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:06 smithi086 ceph-mon[55252]: from='client.44734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:07.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:07 smithi067 ceph-mon[62070]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:07.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:07 smithi067 ceph-mon[64635]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:07.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:07 smithi086 ceph-mon[55252]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:07.943 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:45:07.943 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw": { 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 4 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 17 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-06T12:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:45:08.554 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:08 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:08.555 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:08 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3634420992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:08.555 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:08 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:08.555 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:08 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3634420992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:08.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T12:45:08.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:08 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:08.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:08 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3634420992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:09.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:09 smithi086 ceph-mon[55252]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:09.903 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:09 smithi067 ceph-mon[62070]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:09.903 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:09 smithi067 ceph-mon[64635]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:10.631 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "osd", 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw" 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "17/23 daemons upgraded", 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed.", 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": true 2024-05-06T12:45:10.632 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:45:11.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T12:45:11.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:11 smithi086 ceph-mon[55252]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:11.874 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:11 smithi067 ceph-mon[62070]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:11.874 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:11 smithi067 ceph-mon[64635]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:12.776 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:12 smithi067 ceph-mon[62070]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:12.777 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:12 smithi067 ceph-mon[64635]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:12.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:12 smithi086 ceph-mon[55252]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout: daemon iscsi.foo.smithi067.fnpkny on smithi067 is in unknown state 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi067.fnpkny on host smithi067 failed. 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrade daemon: iscsi.foo.smithi067.fnpkny: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi067.fnpkny ... 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout:Creating ceph-iscsi config... 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout:Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/iscsi-gateway.cfg 2024-05-06T12:45:13.261 INFO:teuthology.orchestra.run.smithi067.stdout:Write file: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/iscsi.foo.smithi067.fnpkny/tcmu-runner-entrypoint.sh 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d847b865c\x2d0ba3\x2d11ef\x2dbc97\x2dc7b262605968.slice/ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl start ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T12:45:13.262 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T12:45:13.263 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1087, in deploy_daemon 2024-05-06T12:45:13.263 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1223, in deploy_daemon_units 2024-05-06T12:45:13.263 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T12:45:13.263 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: systemctl start ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny: Job for ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service failed because the control process exited with error code. 2024-05-06T12:45:13.263 INFO:teuthology.orchestra.run.smithi067.stdout:See "systemctl status ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" and "journalctl -xeu ceph-847b865c-0ba3-11ef-bc97-c7b262605968@iscsi.foo.smithi067.fnpkny.service" for details. 2024-05-06T12:45:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:13 smithi086 ceph-mon[55252]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:13.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:13 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/3853447882' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:45:13.918 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:13 smithi067 ceph-mon[62070]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:13.918 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:13 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/3853447882' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:45:13.919 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:13 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:45:13] "GET /metrics HTTP/1.1" 200 38759 "" "Prometheus/2.33.4" 2024-05-06T12:45:13.919 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:13 smithi067 ceph-mon[64635]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:13.919 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:13 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/3853447882' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T12:45:13.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-05-06T12:45:15.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:15 smithi086 ceph-mon[55252]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:15.893 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:15 smithi067 ceph-mon[62070]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:15.893 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:15 smithi067 ceph-mon[64635]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:16.117 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-05-06T12:45:16.718 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:16 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/380258285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:16.718 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:16 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/380258285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:16.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-05-06T12:45:16.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:16 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/380258285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:17.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:17 smithi067 ceph-mon[64635]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:17.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:17 smithi067 ceph-mon[62070]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:17.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:17 smithi086 ceph-mon[55252]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:18.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:18 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:18.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:18 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:18.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:18 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:18.891 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)" 2024-05-06T12:45:19.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-05-06T12:45:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:19 smithi086 ceph-mon[55252]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:19 smithi086 ceph-mon[55252]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:45:19.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:19 smithi086 ceph-mon[55252]: from='client.? 172.21.15.67:0/425229990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:19.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:19 smithi067 ceph-mon[62070]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:19.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:19 smithi067 ceph-mon[62070]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:45:19.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:19 smithi067 ceph-mon[62070]: from='client.? 172.21.15.67:0/425229990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:19.894 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:19 smithi067 ceph-mon[64635]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:19.894 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:19 smithi067 ceph-mon[64635]: from='mgr.25309 172.21.15.67:0/2607862534' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T12:45:19.895 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:19 smithi067 ceph-mon[64635]: from='client.? 172.21.15.67:0/425229990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T12:45:20.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:20 smithi086 ceph-mon[55252]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:20.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:20 smithi067 ceph-mon[62070]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:20.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:20 smithi067 ceph-mon[64635]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:21.742 INFO:teuthology.orchestra.run.smithi067.stdout:osd 8 91s ago - 2024-05-06T12:45:22.477 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T12:45:22.486 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-06T12:45:22.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-05-06T12:45:23.129 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:22 smithi067 ceph-mon[62070]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:23.129 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:22 smithi067 ceph-mon[62070]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:23.129 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:22 smithi067 ceph-mon[64635]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:23.129 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:22 smithi067 ceph-mon[64635]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:23.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:22 smithi086 ceph-mon[55252]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:23.413 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:22 smithi086 ceph-mon[55252]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:24.037 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:45:23] "GET /metrics HTTP/1.1" 200 38759 "" "Prometheus/2.33.4" 2024-05-06T12:45:25.213 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-06T12:45:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "image": "quay.io/ceph/ceph", 2024-05-06T12:45:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "registry": "quay.io", 2024-05-06T12:45:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "bare_image": "ceph/ceph", 2024-05-06T12:45:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "versions": [] 2024-05-06T12:45:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-06T12:45:25.214 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:24 smithi067 ceph-mon[62070]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:25.214 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:24 smithi067 ceph-mon[62070]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:25.214 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:24 smithi067 ceph-mon[64635]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:25.215 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:24 smithi067 ceph-mon[64635]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:25.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:24 smithi086 ceph-mon[55252]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:25.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:24 smithi086 ceph-mon[55252]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:25.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-05-06T12:45:27.165 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:26 smithi067 ceph-mon[62070]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:27.165 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:26 smithi067 ceph-mon[64635]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:27.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:26 smithi086 ceph-mon[55252]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:28.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:27 smithi067 ceph-mon[62070]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:28.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:27 smithi067 ceph-mon[64635]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:28.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:27 smithi086 ceph-mon[55252]: from='client.44686 -' entity='client.iscsi.foo.smithi067.fnpkny' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T12:45:28.533 INFO:teuthology.orchestra.run.smithi067.stdout: "16.2.0", 2024-05-06T12:45:29.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:28 smithi067 ceph-mon[62070]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:29.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:28 smithi067 ceph-mon[62070]: from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:29.175 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:28 smithi067 ceph-mon[64635]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:29.175 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:28 smithi067 ceph-mon[64635]: from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:29.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-05-06T12:45:29.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:28 smithi086 ceph-mon[55252]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T12:45:29.370 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:28 smithi086 ceph-mon[55252]: from='client.44782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:31.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:30 smithi067 ceph-mon[62070]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:31.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:30 smithi067 ceph-mon[64635]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:31.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:30 smithi086 ceph-mon[55252]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:32.226 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:31 smithi067 ceph-mon[62070]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:32.226 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:31 smithi067 ceph-mon[64635]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:32.369 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:31 smithi086 ceph-mon[55252]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T12:45:32.556 INFO:teuthology.orchestra.run.smithi067.stdout: "v16.2.2", 2024-05-06T12:45:32.556 INFO:teuthology.orchestra.run.smithi067.stdout: "v16.2.2-20210505", 2024-05-06T12:45:33.173 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:32 smithi067 ceph-mon[64635]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:33.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:32 smithi067 ceph-mon[62070]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:33.204 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-06T12:45:33.213 INFO:tasks.cephadm:Teardown begin 2024-05-06T12:45:33.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T12:45:33.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T12:45:33.272 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:32 smithi086 ceph-mon[55252]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T12:45:33.277 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-06T12:45:33.278 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-06T12:45:33.303 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-06T12:45:33.335 INFO:tasks.cephadm:Stopping all daemons... 2024-05-06T12:45:33.335 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-06T12:45:33.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a 2024-05-06T12:45:33.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:33 smithi067 systemd[1]: Stopping Ceph mon.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:33.975 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a[62066]: 2024-05-06T12:45:33.641+0000 7f1c65151700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a[62066]: 2024-05-06T12:45:33.641+0000 7f1c65151700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:33 smithi067 podman[88752]: 2024-05-06 12:45:33.814965261 +0000 UTC m=+0.272665068 container died 60a77a61933a66d5e9d7fcb15c3346f03dd3c96444222cab174b614e87525c7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:45:33] ENGINE Bus STOPPING 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:45:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:45:33] ENGINE Bus STOPPED 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:45:33] ENGINE Bus STARTING 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: ::ffff:172.21.15.86 - - [06/May/2024:12:45:33] "GET /metrics HTTP/1.1" 200 38759 "" "Prometheus/2.33.4" 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: CherryPy Checker: 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: The Application mounted at '' has an empty config. 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:45:33.976 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:45:33] ENGINE Serving on http://:::9283 2024-05-06T12:45:33.977 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: [06/May/2024:12:45:33] ENGINE Bus STARTED 2024-05-06T12:45:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:34 smithi067 podman[88752]: 2024-05-06 12:45:34.167142566 +0000 UTC m=+0.624842381 container cleanup 60a77a61933a66d5e9d7fcb15c3346f03dd3c96444222cab174b614e87525c7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-06T12:45:34.475 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 06 12:45:34 smithi067 bash[88752]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-a 2024-05-06T12:45:34.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.a.service' 2024-05-06T12:45:34.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:45:34.698 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-06T12:45:34.698 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-05-06T12:45:34.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.c 2024-05-06T12:45:34.975 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:34 smithi067 systemd[1]: Stopping Ceph mon.c for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:35.381 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c[64631]: 2024-05-06T12:45:34.993+0000 7f7e344d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:45:35.382 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c[64631]: 2024-05-06T12:45:34.993+0000 7f7e344d6700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-05-06T12:45:35.382 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:35 smithi067 podman[88971]: 2024-05-06 12:45:35.136573164 +0000 UTC m=+0.252779099 container died 120694fd2a6c84d883187c0a36c195f7495f9f062ac2ac47a0b791ebfb448437 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-06T12:45:35.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:35 smithi067 podman[88971]: 2024-05-06 12:45:35.555467324 +0000 UTC m=+0.671673305 container cleanup 120694fd2a6c84d883187c0a36c195f7495f9f062ac2ac47a0b791ebfb448437 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:45:35.725 INFO:journalctl@ceph.mon.c.smithi067.stdout:May 06 12:45:35 smithi067 bash[88971]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-c 2024-05-06T12:45:35.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.c.service' 2024-05-06T12:45:36.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:45:36.018 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-05-06T12:45:36.019 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-05-06T12:45:36.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.b 2024-05-06T12:45:36.320 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:36 smithi086 systemd[1]: Stopping Ceph mon.b for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:36.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b[55248]: 2024-05-06T12:45:36.317+0000 7f2a8eff6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:45:36.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b[55248]: 2024-05-06T12:45:36.317+0000 7f2a8eff6700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-05-06T12:45:36.619 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:36 smithi086 podman[73721]: 2024-05-06 12:45:36.390058099 +0000 UTC m=+0.179653506 container died 3215d78b6d70e2fff1839bfccb375dd3a511698dbd0f049c611601c44a40fff7 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-70ce953) 2024-05-06T12:45:37.119 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:36 smithi086 podman[73721]: 2024-05-06 12:45:36.885048216 +0000 UTC m=+0.674643622 container cleanup 3215d78b6d70e2fff1839bfccb375dd3a511698dbd0f049c611601c44a40fff7 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-06T12:45:37.120 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 06 12:45:36 smithi086 bash[73721]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mon-b 2024-05-06T12:45:37.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mon.b.service' 2024-05-06T12:45:37.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:45:37.303 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-05-06T12:45:37.303 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-06T12:45:37.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y 2024-05-06T12:45:37.619 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:37 smithi067 systemd[1]: Stopping Ceph mgr.y for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:37.920 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:37 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:45:37.617+0000 7f33a973d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:45:37.921 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:37 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y[57803]: 2024-05-06T12:45:37.617+0000 7f33a973d700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-06T12:45:37.921 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:37 smithi067 podman[89091]: 2024-05-06 12:45:37.649614496 +0000 UTC m=+0.151783422 container died c0f8319f1bf4db8b28d13e71d1898213ead589204408228f1abdad83a33c8fc8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953) 2024-05-06T12:45:38.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:38 smithi067 podman[89091]: 2024-05-06 12:45:38.060527814 +0000 UTC m=+0.562696791 container cleanup c0f8319f1bf4db8b28d13e71d1898213ead589204408228f1abdad83a33c8fc8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-06T12:45:38.225 INFO:journalctl@ceph.mgr.y.smithi067.stdout:May 06 12:45:38 smithi067 bash[89091]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-y 2024-05-06T12:45:38.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.y.service' 2024-05-06T12:45:38.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:45:38.471 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-06T12:45:38.471 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-06T12:45:38.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x 2024-05-06T12:45:38.788 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:38 smithi086 systemd[1]: Stopping Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:39.082 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:38 smithi086 podman[73844]: 2024-05-06 12:45:38.787256051 +0000 UTC m=+0.125747738 container died cad655065a4e5282857ab2f04d6dd8654c12396a34a11185a890b09c19838d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:45:39.350 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:39 smithi086 podman[73844]: 2024-05-06 12:45:39.164718382 +0000 UTC m=+0.503210108 container cleanup cad655065a4e5282857ab2f04d6dd8654c12396a34a11185a890b09c19838d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, ceph=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:45:39.350 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:39 smithi086 bash[73844]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x 2024-05-06T12:45:39.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service' 2024-05-06T12:45:39.613 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:39 smithi086 podman[73858]: 2024-05-06 12:45:39.349505471 +0000 UTC m=+0.561307731 container remove cad655065a4e5282857ab2f04d6dd8654c12396a34a11185a890b09c19838d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-06T12:45:39.613 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:39 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:45:39.613 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:39 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-06T12:45:39.614 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:39 smithi086 systemd[1]: Stopped Ceph mgr.x for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:45:39.614 INFO:journalctl@ceph.mgr.x.smithi086.stdout:May 06 12:45:39 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@mgr.x.service: Consumed 20.053s CPU time. 2024-05-06T12:45:39.630 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:45:39.630 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-06T12:45:39.630 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-06T12:45:39.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.0 2024-05-06T12:45:39.916 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:39 smithi067 systemd[1]: Stopping Ceph osd.0 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:40.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:39 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[68006]: 2024-05-06T12:45:39.914+0000 7fb6df796700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:45:40.225 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:39 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[68006]: 2024-05-06T12:45:39.914+0000 7fb6df796700 -1 osd.0 152 *** Got signal Terminated *** 2024-05-06T12:45:40.226 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:39 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0[68006]: 2024-05-06T12:45:39.914+0000 7fb6df796700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:45:44.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:45:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:45:44.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:45:44.726 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:45:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:45:44.238Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:45:45.692 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:45 smithi067 podman[89213]: 2024-05-06 12:45:45.410364627 +0000 UTC m=+5.590202341 container died e278b8d0f47ff549b36bca9049215e18e466224dd015652567bc89019d93c61a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:45:45.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:45 smithi067 podman[89213]: 2024-05-06 12:45:45.765741656 +0000 UTC m=+5.945579405 container cleanup e278b8d0f47ff549b36bca9049215e18e466224dd015652567bc89019d93c61a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5) 2024-05-06T12:45:45.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:45 smithi067 bash[89213]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0 2024-05-06T12:45:46.357 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:46 smithi067 podman[89230]: 2024-05-06 12:45:46.033459132 +0000 UTC m=+0.620363985 container remove e278b8d0f47ff549b36bca9049215e18e466224dd015652567bc89019d93c61a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0, ceph=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-06T12:45:46.699 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:46 smithi067 podman[89321]: 2024-05-06 12:45:46.259595042 +0000 UTC m=+0.018381417 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:45:46.700 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:46 smithi067 podman[89321]: 2024-05-06 12:45:46.359442111 +0000 UTC m=+0.118228502 container create 0904e2e84c017b198de723be62757c8ca42625f7503a5fbe633d7eb6394457e8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:45:46.700 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:46 smithi067 podman[89321]: 2024-05-06 12:45:46.570408014 +0000 UTC m=+0.329194368 container init 0904e2e84c017b198de723be62757c8ca42625f7503a5fbe633d7eb6394457e8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:45:46.700 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:46 smithi067 podman[89321]: 2024-05-06 12:45:46.573725462 +0000 UTC m=+0.332511816 container start 0904e2e84c017b198de723be62757c8ca42625f7503a5fbe633d7eb6394457e8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430) 2024-05-06T12:45:46.700 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:46 smithi067 podman[89321]: 2024-05-06 12:45:46.698676111 +0000 UTC m=+0.457462480 container attach 0904e2e84c017b198de723be62757c8ca42625f7503a5fbe633d7eb6394457e8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-06T12:45:46.975 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 06 12:45:46 smithi067 podman[89321]: 2024-05-06 12:45:46.884676171 +0000 UTC m=+0.643462524 container died 0904e2e84c017b198de723be62757c8ca42625f7503a5fbe633d7eb6394457e8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-06T12:45:47.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.0.service' 2024-05-06T12:45:47.652 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:45:47.652 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-06T12:45:47.652 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-06T12:45:47.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.1 2024-05-06T12:45:47.874 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:47 smithi067 systemd[1]: Stopping Ceph osd.1 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:48.226 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[70474]: 2024-05-06T12:45:48.018+0000 7fef274af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:45:48.226 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[70474]: 2024-05-06T12:45:48.018+0000 7fef274af700 -1 osd.1 152 *** Got signal Terminated *** 2024-05-06T12:45:48.226 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:48 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1[70474]: 2024-05-06T12:45:48.018+0000 7fef274af700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:45:49.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:45:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:45:48.811Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.67:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.67:8765: connect: connection refused" 2024-05-06T12:45:49.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:45:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:45:48.811Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.67:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.67:8765: connect: connection refused" 2024-05-06T12:45:49.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:45:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:45:48.815Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.67:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.67:8765: connect: connection refused" 2024-05-06T12:45:49.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:45:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:45:48.815Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.67:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.67:8765: connect: connection refused" 2024-05-06T12:45:49.120 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:45:48 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:45:48.815Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.67:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.67:8765: connect: connection refused" 2024-05-06T12:45:53.836 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:53 smithi067 podman[89428]: 2024-05-06 12:45:53.532594604 +0000 UTC m=+5.625782766 container died b2661afeacd0cfce448b5768871a0360025ef718f9fb584c05d644860f10b0b8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-06T12:45:54.225 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:53 smithi067 podman[89428]: 2024-05-06 12:45:53.835319625 +0000 UTC m=+5.928507788 container cleanup b2661afeacd0cfce448b5768871a0360025ef718f9fb584c05d644860f10b0b8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-06T12:45:54.225 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:53 smithi067 bash[89428]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1 2024-05-06T12:45:54.225 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:54 smithi067 podman[89458]: 2024-05-06 12:45:54.069787731 +0000 UTC m=+0.533772331 container remove b2661afeacd0cfce448b5768871a0360025ef718f9fb584c05d644860f10b0b8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-05-06T12:45:54.582 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:54 smithi067 podman[89551]: 2024-05-06 12:45:54.32311603 +0000 UTC m=+0.018775204 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:45:54.582 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:54 smithi067 podman[89551]: 2024-05-06 12:45:54.454239084 +0000 UTC m=+0.149898300 container create 20c49abf1a0bb32119f461110482006e1538760f56fb42d1d84c26b24eb0983b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430) 2024-05-06T12:45:54.582 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:45:54 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:45:54.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:45:54.582 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:45:54 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:45:54.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:45:54.583 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:45:54 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:45:54.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:45:54.975 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:54 smithi067 podman[89551]: 2024-05-06 12:45:54.675866632 +0000 UTC m=+0.371525805 container init 20c49abf1a0bb32119f461110482006e1538760f56fb42d1d84c26b24eb0983b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:45:54.976 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:54 smithi067 podman[89551]: 2024-05-06 12:45:54.679168612 +0000 UTC m=+0.374827788 container start 20c49abf1a0bb32119f461110482006e1538760f56fb42d1d84c26b24eb0983b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-05-06T12:45:54.976 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:54 smithi067 podman[89551]: 2024-05-06 12:45:54.784050598 +0000 UTC m=+0.479709799 container attach 20c49abf1a0bb32119f461110482006e1538760f56fb42d1d84c26b24eb0983b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-06T12:45:55.287 INFO:journalctl@ceph.osd.1.smithi067.stdout:May 06 12:45:54 smithi067 podman[89551]: 2024-05-06 12:45:54.987350771 +0000 UTC m=+0.683009945 container died 20c49abf1a0bb32119f461110482006e1538760f56fb42d1d84c26b24eb0983b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-05-06T12:45:55.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.1.service' 2024-05-06T12:45:55.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:45:55.674 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-06T12:45:55.674 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-06T12:45:55.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.2 2024-05-06T12:45:55.975 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:45:55 smithi067 systemd[1]: Stopping Ceph osd.2 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:45:56.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:45:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[72960]: 2024-05-06T12:45:56.023+0000 7fe798869700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:45:56.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:45:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[72960]: 2024-05-06T12:45:56.023+0000 7fe798869700 -1 osd.2 152 *** Got signal Terminated *** 2024-05-06T12:45:56.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:45:56 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2[72960]: 2024-05-06T12:45:56.023+0000 7fe798869700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:46:01.809 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:01 smithi067 podman[89656]: 2024-05-06 12:46:01.537437835 +0000 UTC m=+5.636506578 container died b6fce96724515db7ad7bf7888f1bdc251ba3165be818f39cb3c20da22c25ff2d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_BRANCH=HEAD) 2024-05-06T12:46:02.093 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:01 smithi067 podman[89656]: 2024-05-06 12:46:01.882505364 +0000 UTC m=+5.981574101 container cleanup b6fce96724515db7ad7bf7888f1bdc251ba3165be818f39cb3c20da22c25ff2d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:46:02.093 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:01 smithi067 bash[89656]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2 2024-05-06T12:46:02.475 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:02 smithi067 podman[89678]: 2024-05-06 12:46:02.091987064 +0000 UTC m=+0.551512621 container remove b6fce96724515db7ad7bf7888f1bdc251ba3165be818f39cb3c20da22c25ff2d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-06T12:46:02.849 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:02 smithi067 podman[89770]: 2024-05-06 12:46:02.417804857 +0000 UTC m=+0.018391775 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:46:02.849 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:02 smithi067 podman[89770]: 2024-05-06 12:46:02.51788884 +0000 UTC m=+0.118475732 container create e782664ad00ed69fd666d8b8d1a21bd532d051153fd3f7ff021d5596a544f748 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-06T12:46:02.849 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:02 smithi067 podman[89770]: 2024-05-06 12:46:02.737301174 +0000 UTC m=+0.337888066 container init e782664ad00ed69fd666d8b8d1a21bd532d051153fd3f7ff021d5596a544f748 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-06T12:46:02.850 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:02 smithi067 podman[89770]: 2024-05-06 12:46:02.740574759 +0000 UTC m=+0.341161651 container start e782664ad00ed69fd666d8b8d1a21bd532d051153fd3f7ff021d5596a544f748 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953) 2024-05-06T12:46:02.850 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:02 smithi067 podman[89770]: 2024-05-06 12:46:02.848595611 +0000 UTC m=+0.449182505 container attach e782664ad00ed69fd666d8b8d1a21bd532d051153fd3f7ff021d5596a544f748 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-05-06T12:46:03.225 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:03 smithi067 podman[89770]: 2024-05-06 12:46:03.049837976 +0000 UTC m=+0.650424873 container died e782664ad00ed69fd666d8b8d1a21bd532d051153fd3f7ff021d5596a544f748 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.license=GPLv2) 2024-05-06T12:46:03.655 INFO:journalctl@ceph.osd.2.smithi067.stdout:May 06 12:46:03 smithi067 podman[89796]: 2024-05-06 12:46:03.640979187 +0000 UTC m=+0.585385875 container remove e782664ad00ed69fd666d8b8d1a21bd532d051153fd3f7ff021d5596a544f748 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-2-deactivate, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-05-06T12:46:03.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.2.service' 2024-05-06T12:46:03.702 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:46:03.703 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-06T12:46:03.703 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-06T12:46:03.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.3 2024-05-06T12:46:03.975 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:03 smithi067 systemd[1]: Stopping Ceph osd.3 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:04.475 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[75430]: 2024-05-06T12:46:04.060+0000 7fd2abb7e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:46:04.475 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[75430]: 2024-05-06T12:46:04.060+0000 7fd2abb7e700 -1 osd.3 152 *** Got signal Terminated *** 2024-05-06T12:46:04.475 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3[75430]: 2024-05-06T12:46:04.060+0000 7fd2abb7e700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:46:04.476 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:46:04.238Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:04.476 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:04.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:04.476 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:04 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:04.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:06.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:06.092+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:07.073 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:06 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:06.802+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:07.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:07.072+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:08.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:07.832+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:08.089 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:07 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:07.667+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:08.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:08.087+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:09.053 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:08.685+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:09.054 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:08 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:08.836+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:09.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:09.052+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:09.662 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:09 smithi067 podman[89876]: 2024-05-06 12:46:09.386291201 +0000 UTC m=+5.453153263 container died 7ec6d2a3db7332ab32a4df1b6b40889bc209832ef1bdb7a9f8eaca25d10d8e79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-06T12:46:09.975 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:09 smithi067 podman[89876]: 2024-05-06 12:46:09.760783613 +0000 UTC m=+5.827645669 container cleanup 7ec6d2a3db7332ab32a4df1b6b40889bc209832ef1bdb7a9f8eaca25d10d8e79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-70ce953, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240430) 2024-05-06T12:46:09.975 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:09 smithi067 bash[89876]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3 2024-05-06T12:46:10.018 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:09.704+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:10.019 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:09 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:09.862+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:10.343 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:10 smithi067 podman[89893]: 2024-05-06 12:46:10.01197466 +0000 UTC m=+0.615230245 container remove 7ec6d2a3db7332ab32a4df1b6b40889bc209832ef1bdb7a9f8eaca25d10d8e79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-06T12:46:10.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:10 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:10.017+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:10.649 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:10 smithi067 podman[89980]: 2024-05-06 12:46:10.245905213 +0000 UTC m=+0.018326907 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:46:10.649 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:10 smithi067 podman[89980]: 2024-05-06 12:46:10.354463055 +0000 UTC m=+0.126884724 container create 1710b331b1af1457512c36b91b2eef753e4cf443833b277ac7fb6c92f076f78b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953) 2024-05-06T12:46:10.650 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:10 smithi067 podman[89980]: 2024-05-06 12:46:10.565507135 +0000 UTC m=+0.337928804 container init 1710b331b1af1457512c36b91b2eef753e4cf443833b277ac7fb6c92f076f78b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T12:46:10.650 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:10 smithi067 podman[89980]: 2024-05-06 12:46:10.56902108 +0000 UTC m=+0.341442751 container start 1710b331b1af1457512c36b91b2eef753e4cf443833b277ac7fb6c92f076f78b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:46:10.650 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:10 smithi067 podman[89980]: 2024-05-06 12:46:10.648797257 +0000 UTC m=+0.421218938 container attach 1710b331b1af1457512c36b91b2eef753e4cf443833b277ac7fb6c92f076f78b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:46:10.736 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:10 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:10.428+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:43.647926+0000 front 2024-05-06T12:45:43.647992+0000 (oldest deadline 2024-05-06T12:46:09.448590+0000) 2024-05-06T12:46:10.975 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 06 12:46:10 smithi067 podman[89980]: 2024-05-06 12:46:10.870839467 +0000 UTC m=+0.643261138 container died 1710b331b1af1457512c36b91b2eef753e4cf443833b277ac7fb6c92f076f78b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-05-06T12:46:11.010 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:10 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:10.734+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:11.010 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:10 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:10.860+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:11.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:11.009+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:11.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.3.service' 2024-05-06T12:46:11.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:46:11.499 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-06T12:46:11.499 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-06T12:46:11.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.4 2024-05-06T12:46:11.730 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:11.470+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:43.647926+0000 front 2024-05-06T12:45:43.647992+0000 (oldest deadline 2024-05-06T12:46:09.448590+0000) 2024-05-06T12:46:11.730 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:11 smithi086 systemd[1]: Stopping Ceph osd.4 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:12.004 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:11.729+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:12.005 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:11.870+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:12.005 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:11.808+0000 7fc2368cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:46:12.005 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:11.808+0000 7fc2368cd700 -1 osd.4 152 *** Got signal Terminated *** 2024-05-06T12:46:12.005 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:11 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:11.809+0000 7fc2368cd700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:46:12.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:12.003+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:12.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:12.741+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:12.869 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:12.493+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:43.647926+0000 front 2024-05-06T12:45:43.647992+0000 (oldest deadline 2024-05-06T12:46:09.448590+0000) 2024-05-06T12:46:13.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:12.967+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:13.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:12 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:12.887+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:13.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:13.693+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:13.870 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:13.507+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:43.647926+0000 front 2024-05-06T12:45:43.647992+0000 (oldest deadline 2024-05-06T12:46:09.448590+0000) 2024-05-06T12:46:14.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:13.876+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:14.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:13 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:13.949+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:14.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:14 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:46:14.239Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:14.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:14 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:14.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:14.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:14 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:14.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:14.869 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:14.543+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:43.647926+0000 front 2024-05-06T12:45:43.647992+0000 (oldest deadline 2024-05-06T12:46:09.448590+0000) 2024-05-06T12:46:14.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:14.658+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:15.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:14.905+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:15.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:14.908+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:15.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:14 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:14.908+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:15.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:15.692+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:15.870 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:15.534+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:43.647926+0000 front 2024-05-06T12:45:43.647992+0000 (oldest deadline 2024-05-06T12:46:09.448590+0000) 2024-05-06T12:46:16.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:15.915+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:16.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:15.915+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:16.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:15.873+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:16.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:15 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:15.873+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:16.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:16.716+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:16.870 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:16.535+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:43.647926+0000 front 2024-05-06T12:45:43.647992+0000 (oldest deadline 2024-05-06T12:46:09.448590+0000) 2024-05-06T12:46:16.870 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4[58529]: 2024-05-06T12:46:16.535+0000 7fc22eeb9700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.949464+0000 front 2024-05-06T12:45:52.949513+0000 (oldest deadline 2024-05-06T12:46:15.849304+0000) 2024-05-06T12:46:17.168 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:16.879+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:17.168 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:16.879+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:17.169 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:16.873+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:17.169 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:16 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:16.873+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:17.469 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:17 smithi086 podman[73981]: 2024-05-06 12:46:17.167096068 +0000 UTC m=+5.477901177 container died 2e371a2f9616261e0dd6006914f213ed9297c204627189457448fc67fb26f821 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-06T12:46:17.869 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:17 smithi086 podman[73981]: 2024-05-06 12:46:17.64282436 +0000 UTC m=+5.953629470 container cleanup 2e371a2f9616261e0dd6006914f213ed9297c204627189457448fc67fb26f821 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-06T12:46:17.869 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:17 smithi086 bash[73981]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4 2024-05-06T12:46:17.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:17.692+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:17.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:17.692+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.728694+0000 front 2024-05-06T12:45:52.728585+0000 (oldest deadline 2024-05-06T12:46:16.828652+0000) 2024-05-06T12:46:18.213 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:17 smithi086 podman[73998]: 2024-05-06 12:46:17.894416366 +0000 UTC m=+0.719442062 container remove 2e371a2f9616261e0dd6006914f213ed9297c204627189457448fc67fb26f821 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, ceph=True) 2024-05-06T12:46:18.214 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:17.869+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:18.214 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:17.869+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:18.214 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:17.910+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:18.214 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:17 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:17.910+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:18.465 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:18 smithi086 podman[74090]: 2024-05-06 12:46:18.116096655 +0000 UTC m=+0.017886053 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:46:18.465 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:18 smithi086 podman[74090]: 2024-05-06 12:46:18.220145484 +0000 UTC m=+0.121934864 container create 884d6f622071604f1218dd5c320b9ca68575757cfaf0041b2affd432f3fa541a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux ) 2024-05-06T12:46:18.465 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:18 smithi086 podman[74090]: 2024-05-06 12:46:18.389345057 +0000 UTC m=+0.291134447 container init 884d6f622071604f1218dd5c320b9ca68575757cfaf0041b2affd432f3fa541a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-06T12:46:18.466 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:18 smithi086 podman[74090]: 2024-05-06 12:46:18.392532508 +0000 UTC m=+0.294321892 container start 884d6f622071604f1218dd5c320b9ca68575757cfaf0041b2affd432f3fa541a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-06T12:46:18.466 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:18 smithi086 podman[74090]: 2024-05-06 12:46:18.464126991 +0000 UTC m=+0.365916386 container attach 884d6f622071604f1218dd5c320b9ca68575757cfaf0041b2affd432f3fa541a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-70ce953) 2024-05-06T12:46:18.862 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:18.705+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:18.862 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:18.705+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.728694+0000 front 2024-05-06T12:45:52.728585+0000 (oldest deadline 2024-05-06T12:46:16.828652+0000) 2024-05-06T12:46:18.862 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 06 12:46:18 smithi086 podman[74090]: 2024-05-06 12:46:18.685514901 +0000 UTC m=+0.587304294 container died 884d6f622071604f1218dd5c320b9ca68575757cfaf0041b2affd432f3fa541a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-4-deactivate, ceph=True, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_CLEAN=True) 2024-05-06T12:46:19.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:18.860+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:19.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:18.860+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:19.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:18.956+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:19.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:18 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:18.956+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:19.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.4.service' 2024-05-06T12:46:19.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:46:19.302 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-06T12:46:19.303 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-06T12:46:19.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.5 2024-05-06T12:46:19.501 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:19 smithi086 systemd[1]: Stopping Ceph osd.5 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:19.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:19.644+0000 7fe27ae4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:46:19.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:19.644+0000 7fe27ae4b700 -1 osd.5 152 *** Got signal Terminated *** 2024-05-06T12:46:19.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:19.644+0000 7fe27ae4b700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:46:19.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:19.723+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:19.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:19.723+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.728694+0000 front 2024-05-06T12:45:52.728585+0000 (oldest deadline 2024-05-06T12:46:16.828652+0000) 2024-05-06T12:46:20.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:19.886+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:20.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:19.886+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:20.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:19.950+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:20.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:19 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:19.950+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:20.952 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:20.686+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:20.952 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:20.686+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.728694+0000 front 2024-05-06T12:45:52.728585+0000 (oldest deadline 2024-05-06T12:46:16.828652+0000) 2024-05-06T12:46:20.952 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:20.928+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:20.953 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:20.928+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:21.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:20.950+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:21.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:20 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:20.950+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:22.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:21.938+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:22.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:21.938+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:22.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:21.729+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:22.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:21.729+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.728694+0000 front 2024-05-06T12:45:52.728585+0000 (oldest deadline 2024-05-06T12:46:16.828652+0000) 2024-05-06T12:46:22.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:21.967+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:22.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:21 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:21.967+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:23.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:22.924+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:23.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:22.924+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:23.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:22.767+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:23.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:22.767+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.728694+0000 front 2024-05-06T12:45:52.728585+0000 (oldest deadline 2024-05-06T12:46:16.828652+0000) 2024-05-06T12:46:23.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:22.953+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:23.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:22 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:22.953+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:23.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:23.549Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:23.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:23 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:23.549Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:24.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:23.889+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:24.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:23.889+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:24.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:23.738+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:44.527564+0000 front 2024-05-06T12:45:44.527524+0000 (oldest deadline 2024-05-06T12:46:07.427593+0000) 2024-05-06T12:46:24.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:23.738+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:52.728694+0000 front 2024-05-06T12:45:52.728585+0000 (oldest deadline 2024-05-06T12:46:16.828652+0000) 2024-05-06T12:46:24.120 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5[60868]: 2024-05-06T12:46:23.738+0000 7fe273437700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.630104+0000 front 2024-05-06T12:45:59.630020+0000 (oldest deadline 2024-05-06T12:46:23.729911+0000) 2024-05-06T12:46:24.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:23.911+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:24.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:23.911+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:24.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:23.911+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:24.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:46:24.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:24.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:24.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:24.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:24 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:24.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:25.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:24.901+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:25.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:24.901+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:25.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:25 smithi086 podman[74195]: 2024-05-06 12:46:25.09577404 +0000 UTC m=+5.571447060 container died 13f5da2a25e3c417c05552b6d41c4c5921a5420e4c09128971aa0109e15f1555 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:46:25.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:24.918+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:25.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:24.918+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:25.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:24 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:24.918+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:25.738 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:25 smithi086 podman[74195]: 2024-05-06 12:46:25.470855039 +0000 UTC m=+5.946528056 container cleanup 13f5da2a25e3c417c05552b6d41c4c5921a5420e4c09128971aa0109e15f1555 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-05-06T12:46:25.739 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:25 smithi086 bash[74195]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5 2024-05-06T12:46:26.089 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:25 smithi086 podman[74223]: 2024-05-06 12:46:25.738168335 +0000 UTC m=+0.636496242 container remove 13f5da2a25e3c417c05552b6d41c4c5921a5420e4c09128971aa0109e15f1555 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-06T12:46:26.090 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:25.950+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:26.090 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:25.950+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:26.090 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:25.874+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:26.090 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:25.874+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:26.091 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:25 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:25.874+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:26.369 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:26 smithi086 podman[74316]: 2024-05-06 12:46:26.089009712 +0000 UTC m=+0.109589883 container create 559998640afd1b3af85507b9beefc3d0be6cc88c3f87a487b71307e2afca2d60 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-06T12:46:26.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:26 smithi086 podman[74316]: 2024-05-06 12:46:25.997062889 +0000 UTC m=+0.017643070 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:46:26.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:26 smithi086 podman[74316]: 2024-05-06 12:46:26.316548878 +0000 UTC m=+0.337129048 container init 559998640afd1b3af85507b9beefc3d0be6cc88c3f87a487b71307e2afca2d60 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-06T12:46:26.370 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:26 smithi086 podman[74316]: 2024-05-06 12:46:26.319609621 +0000 UTC m=+0.340189801 container start 559998640afd1b3af85507b9beefc3d0be6cc88c3f87a487b71307e2afca2d60 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430) 2024-05-06T12:46:26.832 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:26 smithi086 podman[74316]: 2024-05-06 12:46:26.391285105 +0000 UTC m=+0.411865288 container attach 559998640afd1b3af85507b9beefc3d0be6cc88c3f87a487b71307e2afca2d60 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:46:26.833 INFO:journalctl@ceph.osd.5.smithi086.stdout:May 06 12:46:26 smithi086 podman[74316]: 2024-05-06 12:46:26.614648979 +0000 UTC m=+0.635229154 container died 559998640afd1b3af85507b9beefc3d0be6cc88c3f87a487b71307e2afca2d60 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-06T12:46:27.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:26.993+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:27.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:26.993+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:27.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:26.993+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:55.280935+0000 front 2024-05-06T12:45:55.280851+0000 (oldest deadline 2024-05-06T12:46:26.481211+0000) 2024-05-06T12:46:27.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:26.831+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:27.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:26.831+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:27.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:26 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:26.831+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:27.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.5.service' 2024-05-06T12:46:27.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:46:27.268 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-06T12:46:27.268 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-06T12:46:27.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.6 2024-05-06T12:46:27.609 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:27 smithi086 systemd[1]: Stopping Ceph osd.6 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:27.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:27.607+0000 7f290add9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:46:27.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:27.607+0000 7f290add9700 -1 osd.6 152 *** Got signal Terminated *** 2024-05-06T12:46:27.869 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:27.607+0000 7f290add9700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:46:28.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:27.869+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:28.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:27.869+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:28.370 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:27 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:27.869+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:28.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:28.020+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:28.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:28.020+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:28.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:28.020+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:55.280935+0000 front 2024-05-06T12:45:55.280851+0000 (oldest deadline 2024-05-06T12:46:26.481211+0000) 2024-05-06T12:46:29.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:28.877+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:29.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:28.877+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:29.369 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:28.877+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:29.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:28.993+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:29.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:28.993+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:29.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:28 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:28.993+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:55.280935+0000 front 2024-05-06T12:45:55.280851+0000 (oldest deadline 2024-05-06T12:46:26.481211+0000) 2024-05-06T12:46:30.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:29.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:30.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:29.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:30.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:29 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:29.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:30.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:30 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:30.000+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:30.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:30 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:30.000+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:30.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:30 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:30.000+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:55.280935+0000 front 2024-05-06T12:45:55.280851+0000 (oldest deadline 2024-05-06T12:46:26.481211+0000) 2024-05-06T12:46:31.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:30 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:30.825+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:31.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:30 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:30.825+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:31.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:30 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:30.825+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:31.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.001+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:31.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.001+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:31.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.001+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:55.280935+0000 front 2024-05-06T12:45:55.280851+0000 (oldest deadline 2024-05-06T12:46:26.481211+0000) 2024-05-06T12:46:31.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.001+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:07.582192+0000 front 2024-05-06T12:46:07.582135+0000 (oldest deadline 2024-05-06T12:46:30.482052+0000) 2024-05-06T12:46:32.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:31.783+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:32.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:31.783+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:32.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:31.783+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:32.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.965+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:41.378675+0000 front 2024-05-06T12:45:41.378594+0000 (oldest deadline 2024-05-06T12:46:06.678498+0000) 2024-05-06T12:46:32.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.965+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:50.680089+0000 front 2024-05-06T12:45:50.680010+0000 (oldest deadline 2024-05-06T12:46:14.179935+0000) 2024-05-06T12:46:32.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.965+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:55.280935+0000 front 2024-05-06T12:45:55.280851+0000 (oldest deadline 2024-05-06T12:46:26.481211+0000) 2024-05-06T12:46:32.120 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:31 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6[63200]: 2024-05-06T12:46:31.965+0000 7f29033c5700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:07.582192+0000 front 2024-05-06T12:46:07.582135+0000 (oldest deadline 2024-05-06T12:46:30.482052+0000) 2024-05-06T12:46:33.096 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:32.812+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:33.097 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:32.812+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:33.097 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:32 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:32.812+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:33.368 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:33 smithi086 podman[74422]: 2024-05-06 12:46:33.095952373 +0000 UTC m=+5.604598380 container died c803277a9f3db9e4c7df13d168711a4a3d5473820e88a0c6afbd8420fa78a9e2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:46:33.620 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:33 smithi086 podman[74422]: 2024-05-06 12:46:33.558270041 +0000 UTC m=+6.066916044 container cleanup c803277a9f3db9e4c7df13d168711a4a3d5473820e88a0c6afbd8420fa78a9e2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-06T12:46:33.620 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:33 smithi086 bash[74422]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6 2024-05-06T12:46:33.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:46:33.549Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:33.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:33.551Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:33.975 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:33 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:33.551Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:34.119 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:33 smithi086 podman[74441]: 2024-05-06 12:46:33.817985321 +0000 UTC m=+0.717224480 container remove c803277a9f3db9e4c7df13d168711a4a3d5473820e88a0c6afbd8420fa78a9e2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:46:34.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:33.814+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:34.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:33.814+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:34.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:33.814+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:34.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:33 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:33.814+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:08.710841+0000 front 2024-05-06T12:46:08.710816+0000 (oldest deadline 2024-05-06T12:46:33.311176+0000) 2024-05-06T12:46:34.369 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:34 smithi086 podman[74532]: 2024-05-06 12:46:34.021567261 +0000 UTC m=+0.017987313 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:46:34.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:34 smithi086 podman[74532]: 2024-05-06 12:46:34.139673538 +0000 UTC m=+0.136093574 container create 630b639d16749ccd560d5d97d3059f8904898dcaa424eda770ce96c3d0db7b53 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:46:34.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:34 smithi086 podman[74532]: 2024-05-06 12:46:34.337803225 +0000 UTC m=+0.334223261 container init 630b639d16749ccd560d5d97d3059f8904898dcaa424eda770ce96c3d0db7b53 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0) 2024-05-06T12:46:34.370 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:34 smithi086 podman[74532]: 2024-05-06 12:46:34.34079563 +0000 UTC m=+0.337215667 container start 630b639d16749ccd560d5d97d3059f8904898dcaa424eda770ce96c3d0db7b53 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-06T12:46:34.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:46:34.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:34.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:34.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:34.725 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:34 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:34.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:34.813 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:34 smithi086 podman[74532]: 2024-05-06 12:46:34.421244669 +0000 UTC m=+0.417664708 container attach 630b639d16749ccd560d5d97d3059f8904898dcaa424eda770ce96c3d0db7b53 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux ) 2024-05-06T12:46:34.813 INFO:journalctl@ceph.osd.6.smithi086.stdout:May 06 12:46:34 smithi086 podman[74532]: 2024-05-06 12:46:34.639904609 +0000 UTC m=+0.636324637 container died 630b639d16749ccd560d5d97d3059f8904898dcaa424eda770ce96c3d0db7b53 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2) 2024-05-06T12:46:35.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:34 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:34.812+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:35.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:34 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:34.812+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:35.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:34 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:34.812+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:35.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:34 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:34.812+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:08.710841+0000 front 2024-05-06T12:46:08.710816+0000 (oldest deadline 2024-05-06T12:46:33.311176+0000) 2024-05-06T12:46:35.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.6.service' 2024-05-06T12:46:35.231 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:46:35.232 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-06T12:46:35.232 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-06T12:46:35.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.7 2024-05-06T12:46:35.561 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 systemd[1]: Stopping Ceph osd.7 for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:35.855 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:35.559+0000 7fb61b673700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T12:46:35.856 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:35.559+0000 7fb61b673700 -1 osd.7 152 *** Got signal Terminated *** 2024-05-06T12:46:35.856 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:35.559+0000 7fb61b673700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T12:46:35.856 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:35.854+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:35.856 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:35.854+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:35.856 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:35.854+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:35.856 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:35 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:35.854+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:08.710841+0000 front 2024-05-06T12:46:08.710816+0000 (oldest deadline 2024-05-06T12:46:33.311176+0000) 2024-05-06T12:46:37.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:36.831+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:37.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:36.831+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:37.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:36.831+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:37.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:36 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:36.831+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:08.710841+0000 front 2024-05-06T12:46:08.710816+0000 (oldest deadline 2024-05-06T12:46:33.311176+0000) 2024-05-06T12:46:38.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:37.849+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:38.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:37.849+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:38.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:37.849+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:38.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:37 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:37.849+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:08.710841+0000 front 2024-05-06T12:46:08.710816+0000 (oldest deadline 2024-05-06T12:46:33.311176+0000) 2024-05-06T12:46:39.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:38.837+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:39.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:38.837+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:39.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:38.837+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:39.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:38 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:38.837+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:08.710841+0000 front 2024-05-06T12:46:08.710816+0000 (oldest deadline 2024-05-06T12:46:33.311176+0000) 2024-05-06T12:46:40.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:39.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-06T12:45:42.507641+0000 front 2024-05-06T12:45:42.507537+0000 (oldest deadline 2024-05-06T12:46:05.407388+0000) 2024-05-06T12:46:40.119 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:39.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-05-06T12:45:51.808838+0000 front 2024-05-06T12:45:51.808937+0000 (oldest deadline 2024-05-06T12:46:15.308714+0000) 2024-05-06T12:46:40.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:39.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-05-06T12:45:59.909901+0000 front 2024-05-06T12:45:59.910016+0000 (oldest deadline 2024-05-06T12:46:23.409869+0000) 2024-05-06T12:46:40.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:39.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.67:6830 osd.3 since back 2024-05-06T12:46:08.710841+0000 front 2024-05-06T12:46:08.710816+0000 (oldest deadline 2024-05-06T12:46:33.311176+0000) 2024-05-06T12:46:40.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:39 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7[65537]: 2024-05-06T12:46:39.844+0000 7fb613c5f700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.86:6804 osd.4 since back 2024-05-06T12:46:16.211929+0000 front 2024-05-06T12:46:16.212003+0000 (oldest deadline 2024-05-06T12:46:39.711974+0000) 2024-05-06T12:46:41.153 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:40 smithi086 podman[74636]: 2024-05-06 12:46:40.884857559 +0000 UTC m=+5.425572035 container died 6c537a696434276822568a543d233f650997770851ea4575ff6c75354de2b339 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-05-06T12:46:41.528 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:41 smithi086 podman[74636]: 2024-05-06 12:46:41.318536638 +0000 UTC m=+5.859251106 container cleanup 6c537a696434276822568a543d233f650997770851ea4575ff6c75354de2b339 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-06T12:46:41.529 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:41 smithi086 bash[74636]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7 2024-05-06T12:46:41.869 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:41 smithi086 podman[74654]: 2024-05-06 12:46:41.528098012 +0000 UTC m=+0.636627578 container remove 6c537a696434276822568a543d233f650997770851ea4575ff6c75354de2b339 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-05-06T12:46:42.249 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:41 smithi086 podman[74744]: 2024-05-06 12:46:41.837757226 +0000 UTC m=+0.018211444 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T12:46:42.249 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:41 smithi086 podman[74744]: 2024-05-06 12:46:41.953954156 +0000 UTC m=+0.134408374 container create df5c7fd84360e83822bf258d3ce5ca8fe1d86540fdfcb094c8d1171c1a3601c1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T12:46:42.249 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:42 smithi086 podman[74744]: 2024-05-06 12:46:42.164410201 +0000 UTC m=+0.344864418 container init df5c7fd84360e83822bf258d3ce5ca8fe1d86540fdfcb094c8d1171c1a3601c1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True) 2024-05-06T12:46:42.249 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:42 smithi086 podman[74744]: 2024-05-06 12:46:42.168040375 +0000 UTC m=+0.348494595 container start df5c7fd84360e83822bf258d3ce5ca8fe1d86540fdfcb094c8d1171c1a3601c1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T12:46:42.249 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:42 smithi086 podman[74744]: 2024-05-06 12:46:42.247707603 +0000 UTC m=+0.428161827 container attach df5c7fd84360e83822bf258d3ce5ca8fe1d86540fdfcb094c8d1171c1a3601c1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-06T12:46:42.619 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:42 smithi086 podman[74744]: 2024-05-06 12:46:42.455951224 +0000 UTC m=+0.636405442 container died df5c7fd84360e83822bf258d3ce5ca8fe1d86540fdfcb094c8d1171c1a3601c1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0) 2024-05-06T12:46:42.993 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 06 12:46:42 smithi086 podman[74770]: 2024-05-06 12:46:42.975747859 +0000 UTC m=+0.514513052 container remove df5c7fd84360e83822bf258d3ce5ca8fe1d86540fdfcb094c8d1171c1a3601c1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-05-06T12:46:42.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@osd.7.service' 2024-05-06T12:46:43.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:46:43.033 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-06T12:46:43.034 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-05-06T12:46:43.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-847b865c-0ba3-11ef-bc97-c7b262605968@prometheus.a 2024-05-06T12:46:43.369 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 systemd[1]: Stopping Ceph prometheus.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.228Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.228Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.228Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.228Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.228Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.228Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.228Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.229Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.229Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.229Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.229Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T12:46:43.370 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a[51122]: ts=2024-05-06T12:46:43.229Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T12:46:43.371 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 podman[74823]: 2024-05-06 12:46:43.239108315 +0000 UTC m=+0.120760636 container died 8dffac3aef3bed9c386062779276327ec254daea6ffa73645080ba096e0ec185 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:43.738 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 podman[74823]: 2024-05-06 12:46:43.553619787 +0000 UTC m=+0.435272109 container cleanup 8dffac3aef3bed9c386062779276327ec254daea6ffa73645080ba096e0ec185 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:43.739 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:May 06 12:46:43 smithi086 bash[74823]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T12:46:43.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-847b865c-0ba3-11ef-bc97-c7b262605968@prometheus.a.service' 2024-05-06T12:46:43.815 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T12:46:43.815 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-05-06T12:46:43.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-06T12:46:43.843 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:46:43.550Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:43.843 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:43.551Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:43.843 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:43 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:43.551Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:44.635 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=error ts=2024-05-06T12:46:44.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:44.636 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:44.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi067.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.67:8443: connect: connection refused" 2024-05-06T12:46:44.636 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:44 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=warn ts=2024-05-06T12:46:44.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-05-06T12:46:45.725 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:45 smithi067 systemd[1]: Stopping Ceph node-exporter.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:45.725 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:45 smithi067 podman[90345]: 2024-05-06 12:46:45.480472297 +0000 UTC m=+0.104114877 container died 347a17f8d7d6dbe88c2b46790dcc4daa85ca2795f133c6d2bedb2778546b7aed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:46.013 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:45 smithi067 podman[90345]: 2024-05-06 12:46:45.803244543 +0000 UTC m=+0.426887122 container cleanup 347a17f8d7d6dbe88c2b46790dcc4daa85ca2795f133c6d2bedb2778546b7aed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:46.014 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:45 smithi067 bash[90345]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a 2024-05-06T12:46:46.014 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:45 smithi067 bash[90371]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter.a" found: no such container 2024-05-06T12:46:46.333 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:46 smithi067 systemd[1]: Stopping Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:46:46.334 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:46 smithi067 podman[90359]: 2024-05-06 12:46:46.01269962 +0000 UTC m=+0.531607117 container remove 347a17f8d7d6dbe88c2b46790dcc4daa85ca2795f133c6d2bedb2778546b7aed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:46.334 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:46 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:46:46.334 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:46 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-06T12:46:46.335 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:46 smithi067 systemd[1]: Stopped Ceph node-exporter.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:46:46.335 INFO:journalctl@ceph.node-exporter.a.smithi067.stdout:May 06 12:46:46 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.a.service: Consumed 3.809s CPU time. 2024-05-06T12:46:46.676 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:46 smithi067 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a[56596]: level=info ts=2024-05-06T12:46:46.332Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-06T12:46:46.676 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:46 smithi067 podman[90424]: 2024-05-06 12:46:46.383686729 +0000 UTC m=+0.158612550 container died 130b40fe12565a875a0106f759b28421cbadeb4882dfdfeef5bed0f03ca40869 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:46.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:46 smithi067 podman[90424]: 2024-05-06 12:46:46.774421491 +0000 UTC m=+0.549347312 container cleanup 130b40fe12565a875a0106f759b28421cbadeb4882dfdfeef5bed0f03ca40869 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:46.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:46 smithi067 bash[90424]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a 2024-05-06T12:46:46.976 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:46 smithi067 bash[90451]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T12:46:47.296 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:47 smithi067 podman[90438]: 2024-05-06 12:46:47.03420779 +0000 UTC m=+0.647078908 container remove 130b40fe12565a875a0106f759b28421cbadeb4882dfdfeef5bed0f03ca40869 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T12:46:47.297 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:47 smithi067 podman[90438]: 2024-05-06 12:46:47.101481427 +0000 UTC m=+0.714352541 volume remove b0ba3f6a87e893c619eff3e33f264a625e7cbb3924506464ffa09bf0bb34de46 2024-05-06T12:46:47.297 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:47 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-06T12:46:47.297 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:47 smithi067 systemd[1]: Stopped Ceph alertmanager.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:46:47.297 INFO:journalctl@ceph.alertmanager.a.smithi067.stdout:May 06 12:46:47 smithi067 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@alertmanager.a.service: Consumed 1.071s CPU time. 2024-05-06T12:47:21.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-06T12:47:22.869 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:22 smithi086 systemd[1]: Stopping Ceph node-exporter.b for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:47:22.869 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:22 smithi086 podman[75178]: 2024-05-06 12:47:22.731720188 +0000 UTC m=+0.118768627 container died f4564f85d02326d870f3569845c24cc9bf746568f85df6c382c75d1b88a78859 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:47:23.356 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 podman[75178]: 2024-05-06 12:47:23.154153024 +0000 UTC m=+0.541201471 container cleanup f4564f85d02326d870f3569845c24cc9bf746568f85df6c382c75d1b88a78859 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:47:23.356 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 bash[75178]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b 2024-05-06T12:47:23.356 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 bash[75204]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter.b" found: no such container 2024-05-06T12:47:23.619 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:23 smithi086 systemd[1]: Stopping Ceph grafana.a for 847b865c-0ba3-11ef-bc97-c7b262605968... 2024-05-06T12:47:23.619 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 podman[75192]: 2024-05-06 12:47:23.35543576 +0000 UTC m=+0.627805020 container remove f4564f85d02326d870f3569845c24cc9bf746568f85df6c382c75d1b88a78859 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T12:47:23.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-06T12:47:23.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-06T12:47:23.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 systemd[1]: Stopped Ceph node-exporter.b for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:47:23.620 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:May 06 12:47:23 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@node-exporter.b.service: Consumed 3.816s CPU time. 2024-05-06T12:47:23.993 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:23 smithi086 ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a[48353]: t=2024-05-06T12:47:23+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-05-06T12:47:23.993 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:23 smithi086 podman[75256]: 2024-05-06 12:47:23.711037039 +0000 UTC m=+0.131938033 container died 3a4e37b2de3aa28e152cb602a8a18aea0993e2f70faa2a781560b5b984f84cc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, description=Ceph Grafana Container, version=8.5, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2) 2024-05-06T12:47:24.335 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:24 smithi086 podman[75256]: 2024-05-06 12:47:24.0834488 +0000 UTC m=+0.504349799 container cleanup 3a4e37b2de3aa28e152cb602a8a18aea0993e2f70faa2a781560b5b984f84cc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, vcs-type=git, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-05-06T12:47:24.335 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:24 smithi086 bash[75256]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a 2024-05-06T12:47:24.335 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:24 smithi086 bash[75292]: Error: no container with name or ID "ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana.a" found: no such container 2024-05-06T12:47:24.620 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:24 smithi086 podman[75273]: 2024-05-06 12:47:24.33446335 +0000 UTC m=+0.621308699 container remove 3a4e37b2de3aa28e152cb602a8a18aea0993e2f70faa2a781560b5b984f84cc1 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-847b865c-0ba3-11ef-bc97-c7b262605968-grafana-a, distribution-scope=public, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, version=8.5, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64) 2024-05-06T12:47:24.620 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:24 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-06T12:47:24.620 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:24 smithi086 systemd[1]: Stopped Ceph grafana.a for 847b865c-0ba3-11ef-bc97-c7b262605968. 2024-05-06T12:47:24.620 INFO:journalctl@ceph.grafana.a.smithi086.stdout:May 06 12:47:24 smithi086 systemd[1]: ceph-847b865c-0ba3-11ef-bc97-c7b262605968@grafana.a.service: Consumed 4.091s CPU time. 2024-05-06T12:47:47.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T12:47:47.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T12:47:47.216 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-06T12:47:47.218 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140/remote/smithi067/crash 2024-05-06T12:47:47.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/crash -- . 2024-05-06T12:47:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-06T12:47:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-05-06T12:47:47.305 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140/remote/smithi086/crash 2024-05-06T12:47:47.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/crash -- . 2024-05-06T12:47:47.343 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-06T12:47:47.344 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-05-06T12:47:47.345 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-06T12:47:47.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T12:47:47.376 INFO:teuthology.orchestra.run.smithi067.stdout:2024-05-06T12:36:44.308566+0000 mon.a (mon.0) 1005 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:47:47.377 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-06T12:47:47.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[SEC\]' /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T12:47:47.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]' /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T12:47:47.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[WRN\]' /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T12:47:47.587 INFO:teuthology.orchestra.run.smithi067.stdout:2024-05-06T12:36:44.308566+0000 mon.a (mon.0) 1005 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T12:47:47.587 INFO:tasks.cephadm:Compressing logs... 2024-05-06T12:47:47.587 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T12:47:47.630 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:: No such file or directory 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.2.log 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log: /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.smpl.smithi067.wpnjcz.log 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.3.log 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.1.log 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.smpl.smithi067.wpnjcz.log: /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.3.log: gzip 76.4% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.smpl.smithi067.wpnjcz.log.gz 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mgr.y.log 2024-05-06T12:47:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.0.log 2024-05-06T12:47:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.c.log 2024-05-06T12:47:47.657 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log 2024-05-06T12:47:47.657 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2024-05-06T12:47:47.657 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.smpl.smithi086.myvasn.log 2024-05-06T12:47:47.657 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-06T12:47:47.657 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log: /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.smpl.smithi086.myvasn.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.7.log 2024-05-06T12:47:47.657 INFO:teuthology.orchestra.run.smithi086.stderr: 76.5% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.smpl.smithi086.myvasn.log.gz 2024-05-06T12:47:47.658 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.0.log: 93.0% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-06T12:47:47.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.5.log 2024-05-06T12:47:47.662 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.4.log 2024-05-06T12:47:47.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.7.log: /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mgr.x.log 2024-05-06T12:47:47.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.6.log 2024-05-06T12:47:47.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi086.frwvbk.log 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.cephadm.log: 87.6% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr: 83.7% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi086.frwvbk.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.b.log 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr: 76.6% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi086.frwvbk.log.gz 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr: 90.5% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-06T12:47:47.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-06T12:47:47.664 INFO:teuthology.orchestra.run.smithi067.stderr: 94.2% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-06T12:47:47.664 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.a.log 2024-05-06T12:47:47.664 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-06T12:47:47.670 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-06T12:47:47.677 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/tcmu-runner.log 2024-05-06T12:47:47.678 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-06T12:47:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.cephadm.log: 90.7% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-06T12:47:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi067.iyetmn.log 2024-05-06T12:47:47.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/tcmu-runner.log: 84.7% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/tcmu-runner.log.gz 2024-05-06T12:47:47.691 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-06T12:47:47.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi067.iyetmn.log: 76.5% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi067.iyetmn.log.gz 2024-05-06T12:47:47.702 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-06T12:47:47.720 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mgr.x.log.gz 2024-05-06T12:47:47.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-06T12:47:47.747 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-06T12:47:47.933 INFO:teuthology.orchestra.run.smithi086.stderr: 92.4% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.b.log.gz 2024-05-06T12:47:48.013 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mgr.y.log.gz 2024-05-06T12:47:48.133 INFO:teuthology.orchestra.run.smithi067.stderr: 92.2% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.c.log.gz 2024-05-06T12:47:48.635 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-mon.a.log.gz 2024-05-06T12:47:48.720 INFO:teuthology.orchestra.run.smithi086.stderr: 93.5% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.6.log.gz 2024-05-06T12:47:48.766 INFO:teuthology.orchestra.run.smithi067.stderr: 93.5% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.2.log.gz 2024-05-06T12:47:48.987 INFO:teuthology.orchestra.run.smithi086.stderr: 93.8% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.7.log.gz 2024-05-06T12:47:49.032 INFO:teuthology.orchestra.run.smithi086.stderr: 93.6% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.5.log.gz 2024-05-06T12:47:49.129 INFO:teuthology.orchestra.run.smithi086.stderr: 93.8% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.4.log.gz 2024-05-06T12:47:49.131 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-05-06T12:47:49.131 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m1.487s 2024-05-06T12:47:49.131 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m5.422s 2024-05-06T12:47:49.131 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.227s 2024-05-06T12:47:49.185 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.1.log.gz 2024-05-06T12:47:49.275 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.0.log.gz 2024-05-06T12:47:49.560 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/847b865c-0ba3-11ef-bc97-c7b262605968/ceph-osd.3.log.gz 2024-05-06T12:47:49.562 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-06T12:47:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m1.919s 2024-05-06T12:47:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m7.721s 2024-05-06T12:47:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.259s 2024-05-06T12:47:49.562 INFO:tasks.cephadm:Archiving logs... 2024-05-06T12:47:49.563 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140/remote/smithi067/log 2024-05-06T12:47:49.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T12:47:50.529 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140/remote/smithi086/log 2024-05-06T12:47:50.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T12:47:51.321 INFO:tasks.cephadm:Removing cluster... 2024-05-06T12:47:51.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 --force 2024-05-06T12:47:51.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 847b865c-0ba3-11ef-bc97-c7b262605968 --force 2024-05-06T12:47:51.916 INFO:tasks.cephadm:Removing cephadm ... 2024-05-06T12:47:51.917 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-06T12:47:51.934 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-06T12:47:51.948 INFO:tasks.cephadm:Teardown complete 2024-05-06T12:47:51.948 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-06T12:47:51.958 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-06T12:47:51.958 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T12:47:51.976 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T12:47:51.988 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-06T12:47:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T12:47:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-06T12:47:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +1988us[+1989us] +/- 56ms 2024-05-06T12:47:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 4 -1391us[-1389us] +/- 56ms 2024-05-06T12:47:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -1026us[-1026us] +/- 67ms 2024-05-06T12:47:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:47:52.001 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-05-06T12:47:52.005 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T12:47:52.006 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-05-06T12:47:52.006 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 +2028us[+2028us] +/- 56ms 2024-05-06T12:47:52.006 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 67 -1397us[-1396us] +/- 56ms 2024-05-06T12:47:52.006 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -990us[ -990us] +/- 67ms 2024-05-06T12:47:52.006 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T12:47:52.006 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-06T12:47:52.015 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-06T12:47:52.017 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-06T12:47:52.025 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-06T12:47:52.113 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-06T12:47:52.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T12:47:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:47:52.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T12:47:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:47:52.238 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-06T12:47:52.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-06T12:47:52.256 INFO:teuthology.task.internal:Duration was 1719.342386 seconds 2024-05-06T12:47:52.256 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-06T12:47:52.265 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-06T12:47:52.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T12:47:52.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T12:47:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T12:47:52.314 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T12:47:52.654 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-06T12:47:52.655 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-05-06T12:47:52.655 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T12:47:52.676 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-05-06T12:47:52.676 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T12:47:52.719 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-06T12:47:52.719 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T12:47:52.721 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T12:47:52.783 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-06T12:47:52.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T12:47:52.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T12:47:53.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-06T12:47:53.319 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-06T12:47:53.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T12:47:53.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T12:47:53.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-06T12:47:53.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T12:47:53.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T12:47:53.409 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-05-06T12:47:53.434 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-05-06T12:47:53.457 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T12:47:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:47:53.506 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T12:47:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T12:47:53.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-06T12:47:53.529 INFO:teuthology.task.internal:Transferring archived files... 2024-05-06T12:47:53.530 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140/remote/smithi067 2024-05-06T12:47:53.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T12:47:53.590 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140/remote/smithi086 2024-05-06T12:47:53.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T12:47:53.627 INFO:teuthology.task.internal:Removing archive directory... 2024-05-06T12:47:53.627 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T12:47:53.630 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T12:47:53.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-06T12:47:53.687 INFO:teuthology.task.internal:Not uploading archives. 2024-05-06T12:47:53.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-06T12:47:53.697 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-06T12:47:53.697 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T12:47:53.699 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T12:47:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 12:47 /home/ubuntu/cephtest 2024-05-06T12:47:53.730 INFO:teuthology.orchestra.run.smithi086.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 12:47 /home/ubuntu/cephtest 2024-05-06T12:47:53.731 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-06T12:47:53.741 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-06T12:47:53.762 INFO:teuthology.nuke:Checking targets against current locks 2024-05-06T12:47:53.791 DEBUG:teuthology.nuke:shortname: smithi067 2024-05-06T12:47:53.791 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T12:47:53.812 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 12:13:35.995517', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZcHTy/FY5NXQv9ysBe429FHFo9gVXuEan7BlMQMgErhgvxAcQ//sJ8UWbs1rSK6GSjqhfwl3FhRURWczZxpXw='} 2024-05-06T12:47:53.815 DEBUG:teuthology.nuke:shortname: smithi086 2024-05-06T12:47:53.815 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T12:47:53.831 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688140', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 12:13:35.997019', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCzZsL+oNnrbNXuhRYF0rXko3EBSy0fL6JAoBBguEnSRJPXqn0x2cnvFqh50dPIflyLEVyQ43z2VULDdSM45WLY='} 2024-05-06T12:47:53.851 INFO:teuthology.orchestra.console.smithi067:Power off 2024-05-06T12:47:53.851 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-06T12:47:53.875 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-05-06T12:47:53.875 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:47:53.891 INFO:teuthology.orchestra.console.smithi086:Power off 2024-05-06T12:47:53.891 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-06T12:47:53.900 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-06T12:47:53.913 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-05-06T12:47:53.913 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:47:53.937 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-05-06T12:47:57.902 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:47:57.938 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:47:58.012 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-06T12:47:58.047 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-05-06T12:48:02.016 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:48:02.048 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:48:02.127 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-06T12:48:02.158 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-05-06T12:48:06.132 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:48:06.159 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T12:48:06.242 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-05-06T12:48:06.268 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-05-06T12:48:06.344 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-05-06T12:48:06.369 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-05-06T12:48:06.473 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1719.3423862457275 failure_reason: '"2024-05-06T12:36:44.308566+0000 mon.a (mon.0) 1005 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-05-06T12:48:06.473 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T12:48:06.556 INFO:teuthology.run:FAIL